CHIME archive writers (VisWriter, HFBWriter, baseband)

The CHIME-era writer stages produce the CHIME correlator archive formats. They are used by the chime_science_run_* configurations and are documented here at summary level; the archive formats themselves are CHIME conventions, processed downstream (e.g. by gossec) into the archive HDF5 format read by caput/ch_util.

Note

This page summarizes the output of lib/stages/BaseWriter.cpp and its subclasses (VisWriter, HFBWriter) plus BasebandWriter and VisSharedMemWriter; update it when changing those stages’ output.

VisWriter and HFBWriter (BaseWriter)

Both derive from BaseWriter, which manages acquisitions: directories under root_path named from the time of the first sample,

<root_path>/<YYYYMMDDTHHMMSSZ>_<instrument_name>_corr/    (VisWriter)
<root_path>/<YYYYMMDDTHHMMSSZ>_<instrument_name>_hfb/     (HFBWriter)

Within an acquisition, files are named by seconds since acquisition start: <sec:08d>_0000 (VisWriter) or hfb_<sec:08d>_0000 (HFBWriter), and roll over after file_length time samples (default 1024). A sliding window (default 20) of recent samples is kept open for late data; acquisitions close after acq_timeout (default 300 s) of inactivity. Every open file has a companion lock file .<name>.lock containing the writer PID; readers should skip locked files.

The on-disk format is selected by file_type:

  • raw / hfbraw — a .meta + .data file pair (see below);

  • hdf5 / hdf5fast — single .h5 files (hdf5fast pre-allocates contiguous datasets and writes via direct I/O, recording the true sample count in a num_time attribute);

  • ring — like raw but a fixed-size ring that overwrites the oldest samples (used for live monitoring buffers).

Raw format. The .meta file is a msgpack-serialized JSON document holding attributes (instrument and version info, e.g. archive_version: NT_3.1.0, weight_type, git_version_tag), index_map (the freq, input, prod or stack, ev, and time axes), an optional reverse_map, and a structure section giving metadata_size, data_size, and the 4 KiB-aligned frame_size, plus nfreq and ntime. The .data file is a dense (time, frequency) grid of slots of frame_size bytes at offset (time_ind * nfreq + freq_ind) * frame_size, each slot containing a 1-byte valid flag, the frame metadata struct, and the frame payload; missing samples remain zero-filled with valid flag 0.

Key configuration (BaseWriter): root_path, file_type (default hdf5fast), file_length, window, acq_timeout, instrument_name (default chime). HFBWriter additionally records the beam and sub-frequency axes (num_frb_total_beams, factor_upchan).

BasebandWriter

BasebandWriter records triggered baseband dumps. Each event/frequency goes to its own raw file:

<root_path>/baseband_raw_<event_id>/baseband_<event_id>_<freq_id>.data

Frames are appended as a 1-byte valid flag, the serialized BasebandMetadata, and the raw baseband payload. An event’s files are closed after dump_timeout (default 60 s) without new data; max_frames_per_second (default 0 = off) throttles the write rate.

VisSharedMemWriter

VisSharedMemWriter is not a file writer: it exports the most recent num_samples visibility integrations into a POSIX shared-memory ring buffer (by default under /dev/shm/<name>) for live consumers. The region holds a small structured header (write count and ring geometry), a per-slot access record protected by a named semaphore sem.<name>, and the ring of frames (valid flag + metadata + data). A write count of 0 signals a graceful writer shutdown.