News: New packages: hasyutils, pyspectra, tnggui; Fs Data Management

• New packages for hasyutils, pyspectra and tnggui have been distributed. Might affect pyspMonitor, TngGui, SardanaMacroExecutor.

• Fs Data Management:

  News:

  • 21.4.2021: C-FEL will use 1.6 PiB of the new server.

  • 20.4.2021: Improved dCache performance, bandwidth about 100 TiB per day, needs further testing.

  • 14.4.2021: GPFS core will by extended, decided to purchase a new server 5.5 PiB, but: C-FEL buys a part of it and an old 900 TiB server ran out of maintenance.

  • 31.3.2021, dCache need better performance, esp. for many small files.

  • 31.3.2021, Science accounts: the registry-2 is needed for self-service password resets and resource management of science accounts. Currently certain platform adaptors are still missing.

  • Science accounts: request for simplifying the procedure. A discussion with IT has started. This problem is related to registry-2.

  • The pass-though mode is in preparation.

  • The pass-though will be enabled for beamtimes greater than X TB. ( first guess for X: 1 TB, AR and JH are investigating).

  • The GPFS core file system will be upgraded in the near future, 5.6 PiB (- 1.5 PiB for C-FEL).

  New demanding detectors forced us to reconsider our data management strategies. In collaboration with IT a note has been prepared proposing several measures which are currently being discussed:

  Summary:

  • the data-on-disk time will be limited to 90 days after the beamtime as ended.

  • the beamline file system will no longer store complete beamtimes. Instead this server will act as a pass-through buffer for data to be moved to the core file system.

  • a substantial upgrades of the FS-related IT infrastructure of the computer center are needed.

  • storage demands may become part of the proposal evaluation

  • maturity of the data analysis pipeline may also be considered during the proposal evaluation

  • the purchase of new detectors has to be reviewed for impact on data volumes.

  Concerns:

  • implement a simplified procedure for creating scientific accounts?

    The reason for this request is of course that the data are quickly moved to the core where they can only (except for exceptions) be accessed from real accounts.

  • can small beamtimes stay in the BL FS?

  • what about commissioning data? Does the 90 days rule also apply?

    The confluence page about the commissioning path can be found here

    Andre: commissioning data are created when groups overlap and when BL scientists align their setup. Commissioning runs are not saved to tape and they are not deleted. To answer the question: No.

  • FLASH: for beamtime > xTB 90 days, for beamtimes < xTB 360 days, with x being in the range 5-10 TB