-
BORA: A Personalized Data Display for Large-scale Experiments
Authors:
Nicholas Tan Jerome,
Suren Chilingaryan,
Timo Dritschler,
Andreas Kopmann
Abstract:
Given the rapid improvement of the detectors at high-energy physics experiments, the need for real-time data monitoring systems has become imperative. The significance of these systems lies in their ability to display experiment status, steer software and hardware instrumentation, and provide alarms, thus enabling researchers to manage their experiments better. However, researchers typically build…
▽ More
Given the rapid improvement of the detectors at high-energy physics experiments, the need for real-time data monitoring systems has become imperative. The significance of these systems lies in their ability to display experiment status, steer software and hardware instrumentation, and provide alarms, thus enabling researchers to manage their experiments better. However, researchers typically build most data monitoring systems as standalone in-house solutions that cannot be reused for other experiments or future upgrades. We present BORA (personalized collaBORAtive data display), a lightweight browser-based monitoring system that supports diverse protocols and is built specifically for customizable visualization of complex data, which we standardize via video streaming. We show how absolute positioning layout and visual overlay background can address the diverse data display design requirements. Using the client-server architecture, we enable support for diverse communication protocols, with the server component responsible for parsing the incoming data. We integrate the Jupyter Notebook as part of our ecosystem to address the limitations of the web-based framework, providing a foundation to leverage scripting capabilities and integrate popular AI frameworks. Since video streaming is a core component of our framework, we evaluate viable approaches to streaming protocols like HLS, WebRTC, and MPEG-Websocket. The study explores the implications for our use case, highlighting its potential to transform data visualization and decision-making processes.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Low-latency Visual Previews of Large Synchrotron Micro-CT Datasets
Authors:
Nicholas Tan Jerome,
Suren Chilingaryan,
Thomas van de Kamp,
Andreas Kopmann
Abstract:
The unprecedented rate at which synchrotron radiation facilities are producing micro-computed (micro-CT) datasets has resulted in an overwhelming amount of data that scientists struggle to browse and interact with in real-time. Thousands of arthropods are scanned into micro-CT within the NOVA project, producing a large collection of gigabyte-sized datasets. In this work, we present methods to redu…
▽ More
The unprecedented rate at which synchrotron radiation facilities are producing micro-computed (micro-CT) datasets has resulted in an overwhelming amount of data that scientists struggle to browse and interact with in real-time. Thousands of arthropods are scanned into micro-CT within the NOVA project, producing a large collection of gigabyte-sized datasets. In this work, we present methods to reduce the size of this data, scaling it from gigabytes to megabytes, enabling the micro-CT dataset to be delivered in real-time. In addition, arthropods can be identified by scientists even after implementing data reduction methodologies. Our initial step is to devise three distinct visual previews that comply with the best practices of data exploration. Subsequently, each visual preview warrants its own design consideration, thereby necessitating an individual data processing pipeline for each. We aim to present data reduction algorithms applied across the data processing pipelines. Particularly, we reduce size by using the multi-resolution slicemaps, the server-side rendering, and the histogram filtering approaches. In the evaluation, we examine the disparities of each method to identify the most favorable arrangement for our operation, which can then be adjusted for other experiments that have comparable necessities. Our demonstration proved that reducing the dataset size to the megabyte range is achievable without compromising the arthropod's geometry information.
△ Less
Submitted 25 November, 2023;
originally announced November 2023.
-
SciTS: A Benchmark for Time-Series Databases in Scientific Experiments and Industrial Internet of Things
Authors:
Jalal Mostafa,
Sara Wehbi,
Suren Chilingaryan,
Andreas Kopmann
Abstract:
Time-series data has an increasingly growing usage in Industrial Internet of Things (IIoT) and large-scale scientific experiments. Managing time-series data needs a storage engine that can keep up with their constantly growing volumes while providing an acceptable query latency. While traditional ACID databases favor consistency over performance, many time-series databases with novel storage engin…
▽ More
Time-series data has an increasingly growing usage in Industrial Internet of Things (IIoT) and large-scale scientific experiments. Managing time-series data needs a storage engine that can keep up with their constantly growing volumes while providing an acceptable query latency. While traditional ACID databases favor consistency over performance, many time-series databases with novel storage engines have been developed to provide better ingestion performance and lower query latency. To understand how the unique design of a time-series database affects its performance, we design SciTS, a highly extensible and parameterizable benchmark for time-series data. The benchmark studies the data ingestion capabilities of time-series databases especially as they grow larger in size. It also studies the latencies of 5 practical queries from the scientific experiments use case. We use SciTS to evaluate the performance of 4 databases of 4 distinct storage engines: ClickHouse, InfluxDB, TimescaleDB, and PostgreSQL.
△ Less
Submitted 8 June, 2022; v1 submitted 20 April, 2022;
originally announced April 2022.
-
Real-time Local Noise Filter in 3D Visualization of CT Data
Authors:
N. Tan Jerome,
Z. Ateyev,
S. Schmelzle,
S. Chilingaryan,
A. Kopmann
Abstract:
Removing noise in computer tomography (CT) data for real-time 3D visualization is vital to improving the quality of the final display. However, the CT noise cannot be removed by straight averaging because the noise has a broadband spatial frequency that is overlapping with the interesting signal frequencies. To improve the display of structures and features contained in the data, we present spatia…
▽ More
Removing noise in computer tomography (CT) data for real-time 3D visualization is vital to improving the quality of the final display. However, the CT noise cannot be removed by straight averaging because the noise has a broadband spatial frequency that is overlapping with the interesting signal frequencies. To improve the display of structures and features contained in the data, we present spatially variant filtering that performs averaging of sub-regions around a central region. We compare our filter with four other similar spatially variant filters regarding entropy and processing time. The results demonstrate significant improvement of the visual quality with processing time still within the millisecond range.
△ Less
Submitted 24 June, 2018;
originally announced July 2018.
