The High Data Rate Processing and Analysis Initiative of the Helmholtz Association in Germany

Abstract

The Helmholtz Association is the largest scientific organization in Germany. It operates all major German research infrastructures involved in research with photons, neutrons, and ions. These are DESY in Hamburg; the Karlsruhe Institute of Technology (KIT); the Research Centre Jülich (FZJ); the Helmholtz Centres in Geesthacht (HZG), Berlin (HZB), and Dresden-Rossendorf (HZDR); and the GSI Centre for research with heavy ions in Darmstadt. In common, all these centers are facing similar challenges related to dramatically increasing data rates and volumes generated with more and more powerful radiation sources together with larger and faster detectors. On the other hand, each center has its own specific portfolio of long-lasting technical expertise in areas like data analysis, information technology, or hardware development. Therefore, it was obvious to address the challenges by acting in concert. This was the main motivation in 2010 for the launch of a joint project among the partners called the “High Data Rate Processing and Analysis Initiative (HDRI).” The initiative is organized into three basic work packages: “Data Management,” “Real-time Data Processing,” and “Data Analysis, Modelling, and Simulation.” The aim is to carry out the development of methods, hardware components, and software for data acquisition, real-time and offline analysis, documentation and archiving, and for remote access to data. The solutions are finally meant to be integrated at the various experimental stations and thus have to be versatile and flexible to cope with the heterogeneous requirements of the different experiments. The claim to create standard solutions makes it mandatory to closely collaborate with large international activities in the field of data handling, like the European PaNdata project (see article in this issue), but also with vendors of detectors, data evaluation software, etc., as well as with corresponding standardization bodies.