News: ICS; pass-through mode; Remote BL Operation; Fs Data Management

• the pass-through mode is a measure to protect the GPFS beamline cluster against being flooded. If enabled, no longer will all data belonging to a beamtime be available on the BL file system until the end of a beam time. The pass-through mode will now be enabled for beamtimes > 10 TiB. Details can be found here

• ICS: a problem in the communication chain has be identified and removed, speeding up response time by a factor of more than 10, read-access. To be investigated, if this solves also write-access.

• Remote BL Operation: the following requirements have be mentioned during a survey and in a BL mgr meeting
  • SardanaMacroExecutor: end-of-macro better signaled to the log widget: 'Job ended ...'.

  • SardanaMacroExecutor: debugged Command: senv ScanFile "[ 'tst.fio', 'tst.spe']"

  • kiosk mode for multiple applications: solution using icewm under investigation, icons, LaVue, SardanaMacroExecutor (pyspMonitor), etc. (JM, MF)

  • extra remote target host (mux) : tests at P09, extra host with own Tango DB, online.xml from experiment, AIO, check macros.

  • gamma portal: DESY should offer ssh + rsync for download.

    Julia created confluence page, see here

  • zoom licences at the beam lines: can be ordered via AMS, create a ticket in FS-EC.

  • scaling problem due to different monitor resolutions: prepare the session on a low resolution monitor, ask users to use high resolution monitors or use browser zoom.

• Fs Data Management:

  News:

  • 2.6.2021: dCache performance further improved, waiting for good test cases.

  • 12.5.2021: Inhouse research data have to stay longer on disk than 90 days after beamtime.

    In discussion: how to identify these data. Door gives us the proposal leader and the PI. There may be other DESY collaborations not falling into these categories. Option: make a declaration via the gamma portal.

  • 12.5.2021: improving the dCache perfomance is still worked on.

  • 21.4.2021: C-FEL will use 1.6 PiB of the new server.

  • 20.4.2021: Improved dCache performance, bandwidth about 100 TiB per day, needs further testing.

  • 14.4.2021: GPFS core will by extended, decided to purchase a new server 5.5 PiB, but: C-FEL buys a part of it and an old 900 TiB server ran out of maintenance.

  • 31.3.2021, dCache need better performance, esp. for many small files.

  • 31.3.2021, Science accounts: the registry-2 is needed for self-service password resets and resource management of science accounts. Currently certain platform adaptors are still missing.

  • Science accounts: request for simplifying the procedure. A discussion with IT has started. This problem is related to registry-2.

  • The pass-though mode is in preparation.

  • The pass-though will be enabled for beamtimes greater than X TB. ( first guess for X: 1 TB, AR and JH are investigating).

  • The GPFS core file system will be upgraded in the near future, 5.6 PiB (- 1.5 PiB for C-FEL).

  New demanding detectors forced us to reconsider our data management strategies. In collaboration with IT a note has been prepared proposing several measures which are currently being discussed:

  Summary:

  • the data-on-disk time will be limited to 90 days after the beamtime as ended.

  • the beamline file system will no longer store complete beamtimes. Instead this server will act as a pass-through buffer for data to be moved to the core file system.

  • a substantial upgrades of the FS-related IT infrastructure of the computer center are needed.

  • storage demands may become part of the proposal evaluation

  • maturity of the data analysis pipeline may also be considered during the proposal evaluation

  • the purchase of new detectors has to be reviewed for impact on data volumes.

  Concerns:

  • implement a simplified procedure for creating scientific accounts?

    The reason for this request is of course that the data are quickly moved to the core where they can only (except for exceptions) be accessed from real accounts.

  • can small beamtimes stay in the BL FS?

  • what about commissioning data? Does the 90 days rule also apply?

    The confluence page about the commissioning path can be found here

    Andre: commissioning data are created when groups overlap and when BL scientists align their setup. Commissioning runs are not saved to tape and they are not deleted. To answer the question: No.

  • FLASH: for beamtime > xTB 90 days, for beamtimes < xTB 360 days, with x being in the range 5-10 TB