ICE: a scalable, low-cost FPGA-based telescope signal processing and networking system
Authors:
K. Bandura,
A. N. Bender,
J. F. Cliche,
T. de Haan,
M. A. Dobbs,
A. J. Gilbert,
S. Griffin,
G. Hsyu,
D. Ittah,
J. Mena Parra,
J. Montgomery,
T. Pinsonneault-Marotte,
S. Siegel,
G. Smecher,
Q. Y. Tang,
K. Vanderlinde,
N. Whitehorn
Abstract:
We present an overview of the 'ICE' hardware and software framework that implements large arrays of interconnected FPGA-based data acquisition, signal processing and networking nodes economically. The system was conceived for application to radio, millimeter and sub-millimeter telescope readout systems that have requirements beyond typical off-the-shelf processing systems, such as careful control…
▽ More
We present an overview of the 'ICE' hardware and software framework that implements large arrays of interconnected FPGA-based data acquisition, signal processing and networking nodes economically. The system was conceived for application to radio, millimeter and sub-millimeter telescope readout systems that have requirements beyond typical off-the-shelf processing systems, such as careful control of interference signals produced by the digital electronics, and clocking of all elements in the system from a single precise observatory-derived oscillator. A new generation of telescopes operating at these frequency bands and designed with a vastly increased emphasis on digital signal processing to support their detector multiplexing technology or high-bandwidth correlators---data rates exceeding a terabyte per second---are becoming common. The ICE system is built around a custom FPGA motherboard that makes use of an Xilinx Kintex-7 FPGA and ARM-based co-processor. The system is specialized for specific applications through software, firmware, and custom mezzanine daughter boards that interface to the FPGA through the industry-standard FMC specifications. For high density applications, the motherboards are packaged in 16-slot crates with ICE backplanes that implement a low-cost passive full-mesh network between the motherboards in a crate, allow high bandwidth interconnection between crates, and enable data offload to a computer cluster. A Python-based control software library automatically detects and operates the hardware in the array. Examples of specific telescope applications of the ICE framework are presented, namely the frequency-multiplexed bolometer readout systems used for the SPT and Simons Array and the digitizer, F-engine, and networking engine for the CHIME and HIRAX radio interferometers.
△ Less
Submitted 22 August, 2016;
originally announced August 2016.
ICE-based Custom Full-Mesh Network for the CHIME High Bandwidth Radio Astronomy Correlator
Authors:
Kevin Bandura,
Jean-Francois Cliche,
Matt Dobbs,
Adam Gilbert,
David Ittah,
Juan Mena Parra,
Graeme Smecher
Abstract:
New generation radio interferometers encode signals from thousands of antenna feeds across large bandwidth. Channelizing and correlating this data requires networking capabilities that can handle unprecedented data rates with reasonable cost. The Canadian Hydrogen Intensity Mapping Experiment (CHIME) correlator processes 8-bits from N=2048 digitizer inputs across 400~MHz of bandwidth. Measured in…
▽ More
New generation radio interferometers encode signals from thousands of antenna feeds across large bandwidth. Channelizing and correlating this data requires networking capabilities that can handle unprecedented data rates with reasonable cost. The Canadian Hydrogen Intensity Mapping Experiment (CHIME) correlator processes 8-bits from N=2048 digitizer inputs across 400~MHz of bandwidth. Measured in $N^2~\times $ bandwidth, it is the largest radio correlator that has been built. Its digital back-end must exchange and reorganize the 6.6~terabit/s produced by its 128 digitizing and channelizing nodes, and feed it to the 256-node spatial correlator in a way that each node obtains data from all digitizer inputs but across a small fraction of the bandwidth (i.e. `corner-turn'). In order to maximize performance and reliability of the corner-turn system while minimizing cost, a custom networking solution has been implemented. The system makes use of Field Programmable Gate Array (FPGA) transceivers to implement direct, passive, full-mesh, high speed serial connections between sixteen circuit boards in a crate, to exchange data between crates, and to offload the data to a cluster of 256 graphics processing unit (GPU) nodes using standard 10~Gbit/s Ethernet links. The GPU nodes complete the corner-turn by combining data from all crates and then computing visibilities. Eye diagrams and frame error counters confirm error-free operation of the corner-turn network in both the currently operating CHIME Pathfinder telescope (a prototype for the full CHIME telescope) and a representative fraction of the full CHIME hardware providing an end-to-end system validation.
An analysis of an equivalent corner-turn system built with Ethernet switches instead of custom passive data links is provided.
△ Less
Submitted 15 August, 2016;
originally announced August 2016.
