Goals:
- Data acquisition copes with detector speed.
- Data quality is ensured by near real-time analysis.
- DESY supports reconstruction, modelling, simulation.
- Raw and processed data are fully described.
- An Open Access strategy will be implemented.
FS IT challenges for the next 10 - 15 years:
- Scientific computing
- Develope applications for reconstruction, modelling, simulation and near real-time analysis.
- Combine experiment with advanced simulation.
- Establish high throughput data analysis for near real-time decision making, possibly rejecting data.
- Prepare a user analysis environment: science accounts, industrial user accounts, container, portal access
- Collaborate on analysis and compression algorithms and data formats.
- Group of experts provides help with efficient file storage, parallelisation, streaming data processing,
efficient computation on CPUs and GPUs, and so on.
- Data transfer
- Rates: 2015: 250 TB, 2016: 500 TB. Assumption: data rate doubles every year.
- Network: currently 1/10 GE infrastructur, upgrade to 40 GE, etc.
- Optimize transfer rates using streaming protocols (ZMQ).
- Compression
- Involve detector manufacturers for streaming and compression
- Data storage
- Currently about 1 PB
- Implement 'cold data' storage service.
- Data policy: open access after embargo period, total lifetime 10 years, implement
open access operation
- Improve data formats.
- Manage metadata.
- Implement DOIs.
- Compute resources
- CPU, GPU, FPGA
- Grow proportional to data volumes
- Allocate compute resources to beamtimes
- Establish a competence team to optimize utilization of compute resources.
- Cloud computing
- Configuration
- Port analysis code
- Portal
- Support data discoveries and downloads.
- Manage metadata.
- Support web based data processing.
- Cooperation
- Foster common developments with XFEL
- Experiment control area (FS intern)
- Experiment control software
- Fast data acquisition software
- User interfaces
