ASDF and Zarr frame dumps (asdfFileWrite, gdalFileWrite)

Two sibling stages stream kotekan buffers to self-describing scientific formats, one file (or directory) per frame:

  • asdfFileWrite (lib/stages/asdfFileWrite.cpp) writes ASDF files and accepts CHORD or N2 metadata;

  • gdalFileWrite (lib/stages/gdalFileWrite.cpp) writes Zarr v3 stores through GDAL’s multidimensional API and accepts CHORD metadata only.

Note

This page documents the on-disk formats produced by lib/stages/asdfFileWrite.cpp and lib/stages/gdalFileWrite.cpp; update it when changing the entries or attributes written there.

Both share the naming scheme (base_dir is created if missing):

<base_dir>/[<hostname>_]<file_name>.<frame:08d>.asdf    (ASDF)
<base_dir>/[<hostname>_]<file_name>.<frame:08d>.zarr    (Zarr; a directory)

with an 8-digit frame counter and the hostname prefix controlled by prefix_hostname (default true). Common configuration: base_dir, file_name, prefix_hostname, max_frames (default -1 = unlimited; on reaching it the stage shuts kotekan down), and skip_writing (consume without writing).

ASDF files (asdfFileWrite)

The ASDF tree contains one group keyed by the 8-digit frame number (e.g. "00000001"). Array blocks are zlib-compressed (level 9). Within the group, three entries are always present:

  • telescope_name (string), seq_length_nsec (int, FPGA tick length in ns), gps_time_enabled (bool).

For CHORD-metadata buffers the group additionally contains:

  • the frame’s n-dimensional data array, keyed by the buffer name with the host_ prefix and _buffer suffix stripped (e.g. host_upchan_buffer becomes upchan), with the buffer’s element type and dimensions;

  • chord_metadata_version (int[2], currently {1, 0}); name and type (ndarray name and element type string); dim_names (string sequence);

  • if present in the metadata: coarse_freq, freq_upchan_factor, freq_upchan_index (int sequences, one entry per frequency); fpga_seq_num and fpga_seq_time_nsec (first FPGA sequence number and its instrument time in ns); time_downsampling_fpga.

For N2-metadata buffers (single-frequency visibility frames; see N2 visibility files (hdf5N2Write) for field semantics) the group contains inline arrays vis (complex64, \(N_p\)), weights (float32, \(N_p\)), flags (float32, \(N_e\)), eval (float32, \(N_{ev}\)), evec (complex64, \(N_{ev} \times N_e\)), and gain (complex64; due to a bug at the time of writing this entry contains a copy of the eigenvector data, not the gains), plus scalars emethod, erms (stored as an integer), n_valid_fpga_ticks, num_elements, num_prod, freq_id.

Zarr stores (gdalFileWrite)

Each frame becomes a Zarr v3 store (a directory) holding one multidimensional array named after the buffer metadata’s ndarray name, with dimension names taken from the metadata. Arrays are compressed with Blosc (zstd codec, level 9, bit-shuffle); chunks default to the frame extent, with the first dimension split if a chunk would exceed 1 GB.

Root-group attributes mirror the ASDF entries: telescope_name, seq_length_nsec, gps_time_enabled (stored as int), name, type, chord_metadata_version, and, when present in the metadata, coarse_freq, freq_upchan_factor, freq_upchan_index, fpga_seq_num, fpga_seq_time_nsec, time_downsampling_fpga.

The stage rejects N2-metadata buffers.