PaNdata: Open Data Infrastructure for Photon and Neutron Sources

Abstract

Today's scientific research is conducted not just by single experiments, but rather by sequences of related experiments or projects linked by a common theme that lead to a greater understanding of the physical world. This is particularly true of research at large-scale facilities such as synchrotrons, neutron sources, and free-electron lasers. These provide a wide variety of techniques, instruments, and scientific applications as services to the user communities. The common denominator of the different kinds of experiments undertaken at facilities is the need to manage, visualize, and analyze vast amounts of data with high accuracy and reliability in near- to real time. This poses significant challenges on the underlying e-infrastructures at the facilities. The data volumes (up to 10 PB per year), peak data rates, the extremely large number of files of varying sizes and formats, and the unpredictable access patterns pose severe demands on the data infrastructures.