Application of Analysis on the Wire to Streaming NSLS-II Beamline Data
Citation
Diwan, R.; Rizvi, A.; Bhattacharyya, S.; Katramatos, D.; Yager, K.G. "Application of Analysis on the Wire to Streaming NSLS-II Beamline Data
"
New York Scientific Data Summit 2019,
18 8538955.
doi: 10.1109/NYSDS.2018.8538955Summary
We explore the possibility of performing certain kinds of data analysis relevant at synchrotron scattering beamlines 'on the wire'; that is, doing the computation within the network infrastructure itself.
Abstract
The National Synchrotron Light Source II (NSLSII) is a scientific facility that generates vast amounts of many different types of data, including 2D images produced by X-ray scattering experiments. These images are captured and sent from the detectors at rates that usually range from several tens to several hundreds of frames per second. Such high incoming data rates can be difficult to keep up with if real/near-real time image analysis is required. We propose to use the Analysis on Wire framework to introduce intermediate processing while these images are in transit from the detector to their destination. As the 2D images come out of the detector, we capture the corresponding raw data packets while they travel through the network. We use TCP stream reconstruction to correctly reassemble the image data and subsequently execute the analysis code in parallel to analyze multiple images simultaneously. Our measurements indicate that the approach is feasible and effective.
