FS-IT Upgrade Plans

IT and FS-EC are currently planning an upgrade of the FS-IT infrastructure. The reasons are:

• New dectors are constantly brought to Max v. Laue.
• Petra-Extension BLs become operational
• Flash plans to follow the Petra III path
• Other scientific groups/projects want to join.

Issues:

• Create new 'facilities' (like Petra-3) representing scientific groups or projects: Flash, NanoLab and SpecialInstruments (container for e.g. fs-ds VBLs). Goals: organize data storage, accounting, etc.

• Data storage, history and future:
  • 2014: 70 TB
  • 2015: 250 TB (40 TB/Monat)
  • 2016: 600 TB (70 TB/Monat)
  • 2017: 1.2 PB
  • 2018: 2.5 PB
  • 2019: 5 PB
  • 2020: 10 PB

  We assume that the data rate doubles every year.

• GPFS need an extension of the Flash memory:
  • the BL cluster is flooded with many files
  • the core CORE cluster has to deal with many meta data.

• Archive: dCache vs. object store (or a combination).

  Object store:

  • modern technology, used by really big players.
  • highly scalable, up to 100 PB
  • no read-modify-write, but object versioning.
  • data integrity guaranteed.
  • interface: rest api (http)

  Issues:

  • When do we remove files from GPFS? The decision will be based on the 'temperature'.
  • Data policy: How long will we preseve files?
  • How fast should 'old' data be staged?
  • How is the interface?
  • Solution should be compliant with XFEL and CFEL.

• New proxy nodes strategy: monitor the workload on the existing servers and install new, if necessary.

• Compute nodes: scale with data volumes?

• Network for Flash

• Network for PEX-East still in discussion

Time line: plans should be finished in autumn.

Please inform FS-EC, if you have additional requirements.