- ...
data.1.1
- Additional comments: FAT32 is usually supported on Win,Linux and MAC OS; NTFS(MBR)
is supported by Linux and Windows XP/7. Hard drives with capacities larger than 2.2TB as well
as nowadays available pre-formatted drives are set up as NTFS(GPS) drives. This set up is
supported by Windows 7. Such drives usually work on SL 6.x systems available, but are NOT
supported on SL 5.x systems as well as on Windows XP systems (only read support at best).
USB 3 is NOT supported on SL 5.x systems. On Linux systems the external drive needs to
be connected to a system with keyboard/screen login, otherwise the automounter won't work.
Depending on filesystem, additional limitations may exist (e.g. limits for single file size
on FAT or file attributes).
The transfer rates, i.e. the amount of time needed to copy data, is NOT only defined by the
interface definition, which gives the upper theoretical limit of the bandwidth. It highly
depends for example on the hardware itself, the file sizes of the data to transfer, the number
of files, or the filesystem of the storage device. For example: The USB 2.0 protocol has a
theoretical bandwidth of 480 MBit/s (= 60MiB/s). Due to protocol overhead, chipset limitations
etc. the effective bandwidth turns out on the order of
% of the theoretical
bandwidth (
MiB/s) at best. Typical data rates are
% lower than theroy
(
MiB/s) and in particular when copying some ten thousands or even hundreds of
thousands of files of small size (some ten to 100kb per file) the transfer rate will most
likely result much closer to 1MiB/s.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ... III1.2
- Photon
Science user operation ended in Oct. 2012
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ...
1.3
- More precisely, a
beamtime application ID is created when an application for beamtime has been submitted via DOOR.
The associated meta data 'specific period' and 'participants' (who initially will have read
permission to the data set) are created by scheduling the beamtime (beamline manager) and
by providing the names participant names (declaration by the user via DOOR), respectivley.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ... instance1.4
- More precisely it creates a file in the
traditional dCache with tape back end.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ... /path/to/file.tar1.5
- The tarball will be extracted into directory /to/
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ...
'j').1.6
- Please note that other compression schemes might
work as well, depending on system setup etc. Usually 'gzip' ('z'-option) and 'compress' ('Z'-option)
are supported and allow faster compression compared to bzip2, but both schemes do not really allow for
partial archive recovery in case of archive corruption.
Note A.R.: DESY-IT has to tell what 'J' currently means ('J' has been reassigned to xz in GNU tar 2009 (?), before 'J'-option denoted
lzma(?)). Can we assume that -xz, -lzip,-lzma or -lzop are supported by default? As far as
I've realised, tar setup / staging is executed with '-no-auto-compress' and thus do not uses
file suffix to determine the compression program. Do we have to define a file extension for
compressed tarballs? '.tbz' or '.tbz2' are common for bzip2 compression.
Do we have to
explicitly highlight that one should not use the -use-compress-program=PROG option, i.e. with 'zip'
program - depending on zip program / zip format implementation installed, there are
several limitations regarding archive length, number of files .... Btw. this is also the case
for POSIX tar implementation, which we hopefully do not find any more at DESY.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ... migration1.7
- When
migrating from other sources, e.g. data of DESY staff taken at external facilities, one has
to account for the underlying file system [for example ACL settings in AFS). Inventing
ACLs/pnfs 4.1 for the OnlineFS is under discussion, but would make it necessary to set up
OnlineFS from scratch [i.e. fully copy/delete/restore all data from OnlineFS]. Note: Even
with ACLs on the OnlineFS we still have the problem writing data from Pilatus detectors to
it - The first generation Pilatus detectors have a local account 'det' which is not
covered by kerberos or something similar.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ... themself1.8
- Until now, FS-EC (i.e. J. Grabitz and A. Rothkirch) migrated
the data to the dCache in coordination with beamline responsibles.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ... '/scratch'1.9
- When using /scratch as storage space
one has to remind that one writes to the local disk of a specific computer, i.e.
one has to remember on which work group server processing was done and data was written
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ... system3.1
- Take over is limited to a single system, i.e. if both
heads 1 and 2 fail, the attached storage is offline.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
- ... aggregates3.2
- More precisely, the OnlineFS
has eight aggregates, four management aggregates and four aggregates for the data.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.