Data migration to dCache

The invention of new generation detectors at high flux sources leads to huge amounts of data which implies several challenges to data management. Concepts accounting for data archiving, data processing and analysis, format definitions and data organization / structuring or the provision of meta-data demand for development and realization, see also `Working with large-scale facilities and the smallest particles: Research with photons, neutrons and ions' http://www.helmholtz.de/en/research/structure_of_matter/research_with_photons_neutrons_and_ions_pni/ or `HDRI - High Data Rate Processing and Analysis Initiative' http://www.pni-hdri.de/index_eng.html.

The dCache realizes, beside others, parts of current and future data management tasks for DESY Photon Science. It provides tape and disc pools allowing for data archiving and long term data analysis. Within the data management concept, data migration tools are provided to store beamline user as well as in-house data in DESY traditional dCache (with tape backend) and to create meta-data in a catalogue. Stored data can be restored to a disc only instance in dCache via a data portal allowing for data analysis on a mid to long term time scale.

Given the huge amounts of data resulting from even a single experiment, hardware limitations in e.g. capacity, different needs in data reading/writing or access times, rights management or meta-data creation, one has to establish common rules when dealing with data. In the following, descriptions and technical background informations are given on how to use the dCache as well as other resources involved.