-
Observers' Data Access Portal: Realtime Streaming for Astronomical Data
Authors:
T. Coda,
T. Oluyide,
M. S. Lynn,
J. A. Mader,
G. Bruce Berriman,
M. Brodheim,
C. R. Gelino,
J. Good
Abstract:
The W. M. Keck Observatory Archive (KOA) has released the Observers Data Access Portal (ODAP), a web-application that delivers astronomical data from the W. M. Keck Observatory to the scheduled program's principal investigator and their collaborators anywhere in the world in near real-time. Data files and their associated metadata are streamed to a user's desktop machine moments after they are wri…
▽ More
The W. M. Keck Observatory Archive (KOA) has released the Observers Data Access Portal (ODAP), a web-application that delivers astronomical data from the W. M. Keck Observatory to the scheduled program's principal investigator and their collaborators anywhere in the world in near real-time. Data files and their associated metadata are streamed to a user's desktop machine moments after they are written to disk and archived in KOA. The ODAP User Interface is built in React and uses the WebSocket protocol to stream data between KOA and the user. This document describes the design of the tool, challenges encountered, shows how ODAP is integrated into the Keck observing model, and provides an analysis of usage metrics.
△ Less
Submitted 13 September, 2024;
originally announced September 2024.
-
Integration of Data Reduction and Near Real-Time Archiving into the Keck Observing Model
Authors:
Max Brodheim,
John O'Meara,
Jeffrey A. Mader,
G. Bruce Berriman,
Matthew Brown,
Lucas Furhman,
Tyler Tucker,
Christopher R. Gelino,
Meca S. Lynn,
Melanie A. Swain
Abstract:
The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory's Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw…
▽ More
The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory's Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw frames and reduced data in near real-time. These data will be made available to observers and collaborators immediately upon ingestion through a dedicated new interface that will support collaboration and sharing among teams, as well as stream data directly to personal computers without access to WMKO's internal networks. Both the raw and science-ready data products will be made publicly available upon the expiration of data protections.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
The Observers' Data Access Portal at the Keck Observatory Archive
Authors:
T. Oluyide,
M. S. Lynn,
T. Coda,
G. B. Berriman,
M. Brown,
L. Fuhrman,
C. Gelino,
J. Good,
J. Hayashi,
C. -H. Lee,
J. Mader,
M. A. Swain
Abstract:
For all active instruments, the Keck Observatory Archive (KOA) now ingests raw data from the Keck Telescopes within 1 minute of acquisition, quick-look reduced data within 5 minutes of creation, and science ready reduced data for four instruments as they are created by their automated pipelines. On August 1, 2023, KOA released the Observers Data Access Portal (ODAP), which enables observers at the…
▽ More
For all active instruments, the Keck Observatory Archive (KOA) now ingests raw data from the Keck Telescopes within 1 minute of acquisition, quick-look reduced data within 5 minutes of creation, and science ready reduced data for four instruments as they are created by their automated pipelines. On August 1, 2023, KOA released the Observers Data Access Portal (ODAP), which enables observers at the telescope and their collaborators anywhere in the world to securely monitor and download science, calibration, and quick-look data as they are ingested into the archive. The portal is built using Python Socket IO.WebSockets that ensure metadata appear in the portal as the data themselves are ingested. The portal itself is a dynamic web interface built with React. It enables users to view and customize metadata fields, filter metadata according to data type, and download data as they are ingested or in bulk through wget scripts. Observers have used the ODAP since its release and have provided feedback that will guide future releases.
△ Less
Submitted 6 February, 2024;
originally announced February 2024.
-
Real-time Data Ingestion at the Keck Observatory Archive (KOA)
Authors:
G. Bruce Berriman,
M. Brodheim,
M. Brown,
L. Fuhrman,
C. R. Gelino,
M. Kong,
C. -H. Lee,
M. S. Lynn,
J. Mader,
T. Oluyide,
M. A. Swain,
T. Tucker,
A. Laity,
J. Riley
Abstract:
Since February of this year, KOA began to prepare, transfer, and ingest data as they were acquired in near-real time; in most cases data are available to observers through KOA within one minute of acquisition. Real-time ingestion will be complete for all active instruments by the end of Summer 2022. The observatory is supporting the development of modern Python data reduction pipelines, which when…
▽ More
Since February of this year, KOA began to prepare, transfer, and ingest data as they were acquired in near-real time; in most cases data are available to observers through KOA within one minute of acquisition. Real-time ingestion will be complete for all active instruments by the end of Summer 2022. The observatory is supporting the development of modern Python data reduction pipelines, which when delivered, will automatically create science-ready data sets at the end of each night for ingestion into the archive. This presentation will describe the infrastructure developed to support real-time data ingestion, itself part of a larger initiative at the Observatory to modernize end-to-end operations.
During telescope operations, the software at WMKO is executed automatically when a newly acquired file is recognized through monitoring a keyword-based observatory control system; this system is used at Keck to execute virtually all observatory functions. The monitor uses callbacks built into the control system to begin data preparation of files for transmission to the archive on an individual basis: scheduling scripts or file system related triggers are unnecessary. An HTTP-based system called from the Flask micro-framework enables file transfers between WMKO and NExScI and triggers data ingestion at NExScI. The ingestion system at NEXScI is a compact (4 KLOC), highly fault-tolerant, Python-based system. It uses a shared file system to transfer data from WMKO to NExScI. The ingestion code is instrument agnostic, with instrument parameters read from configuration files. It replaces an unwieldy (50 KLOC) C-based system that had been in use since 2004.
△ Less
Submitted 5 December, 2022;
originally announced December 2022.
