News: IT troubles; BL responsibles (IT); Try-Except in macros; small dirs to remain on disk, Remote BL Operation; Fs Data Management

• 18.5.2021: HW and SW upgrade of the firewall, WAN router SW upgrade, several services affected, back to full operation 16:00 h, current state not 100% stable, fail-overs in the firewall cluster, service call open, hope the system gets back to stable operation without reboot.

• BL responsibles (IT) : IT asks for a list of persons whom they could contact in case of IT-related issues, mostly GPFS.

  So far 3 beamlines reported.

• try-except in Macros, P08: stopMacro and abortMacro were caught, stopMacro appeared only once.

• request by P10: gpfs, leave small dirs with few files on disk (not migrate to dCache).

  What about reduced data, factor 1000.

• Remote BL Operation: the following requirements have be mentioned during a survey and in a BL mgr meeting
  • *** kiosk mode for multiple applications: solution using icewm under investigation, icons, LaVue, SardanaMacroExecutor (pyspMonitor), etc. (JM, MF)

  • *** extra remote target host (mux) : tests at P09, extra host with own Tango DB, online.xml from experiment, AIO, check macros.

  • *** gamma portal: DESY should offer ssh + rsync for download.

    Julia created confluence page, see here

  • zoom licences at the beam lines: can be ordered via AMS, create a ticket in FS-EC.

  • scaling problem due to different monitor resolutions: prepare the session on a low resolution monitor, ask users to use high resolution monitors or use browser zoom.

• Fs Data Management:

  News:

  • 12.5.2021: Inhouse research data have to stay longer on disk than 90 days after beamtime.

    In discussion: how to identify these data. Door gives us the proposal leader and the PI. There may be other DESY collaborations not falling into these categories. Option: make a declaration via the gamma portal.

  • 12.5.2021: improving the dCache perfomance is still worked on.

  • 21.4.2021: C-FEL will use 1.6 PiB of the new server.

  • 20.4.2021: Improved dCache performance, bandwidth about 100 TiB per day, needs further testing.

  • 14.4.2021: GPFS core will by extended, decided to purchase a new server 5.5 PiB, but: C-FEL buys a part of it and an old 900 TiB server ran out of maintenance.

  • 31.3.2021, dCache need better performance, esp. for many small files.

  • 31.3.2021, Science accounts: the registry-2 is needed for self-service password resets and resource management of science accounts. Currently certain platform adaptors are still missing.

  • Science accounts: request for simplifying the procedure. A discussion with IT has started. This problem is related to registry-2.

  • The pass-though mode is in preparation.

  • The pass-though will be enabled for beamtimes greater than X TB. ( first guess for X: 1 TB, AR and JH are investigating).

  • The GPFS core file system will be upgraded in the near future, 5.6 PiB (- 1.5 PiB for C-FEL).

  New demanding detectors forced us to reconsider our data management strategies. In collaboration with IT a note has been prepared proposing several measures which are currently being discussed:

  Summary:

  • the data-on-disk time will be limited to 90 days after the beamtime as ended.

  • the beamline file system will no longer store complete beamtimes. Instead this server will act as a pass-through buffer for data to be moved to the core file system.

  • a substantial upgrades of the FS-related IT infrastructure of the computer center are needed.

  • storage demands may become part of the proposal evaluation

  • maturity of the data analysis pipeline may also be considered during the proposal evaluation

  • the purchase of new detectors has to be reviewed for impact on data volumes.

  Concerns:

  • implement a simplified procedure for creating scientific accounts?

    The reason for this request is of course that the data are quickly moved to the core where they can only (except for exceptions) be accessed from real accounts.

  • can small beamtimes stay in the BL FS?

  • what about commissioning data? Does the 90 days rule also apply?

    The confluence page about the commissioning path can be found here

    Andre: commissioning data are created when groups overlap and when BL scientists align their setup. Commissioning runs are not saved to tape and they are not deleted. To answer the question: No.

  • FLASH: for beamtime > xTB 90 days, for beamtimes < xTB 360 days, with x being in the range 5-10 TB