N² visibility files (hdf5N2Write)¶

This page describes the layout and contents of the HDF5 visibility files produced by the kotekan hdf5N2Write stage: every dataset and attribute, with its shape, HDF5 data type, axis labels, and meaning. Array sizes are given symbolically; their values are set by the stage and buffer configuration. For a worked example with concrete values, see the deployment page below.

CHORD pathfinder visibility files

Note

This page documents the on-disk format produced by lib/stages/hdf5N2Write.cpp. If you add, remove, or change any dataset or attribute in that stage, update this page to match (see the note at the top of N2FileData::_open_or_create_file).

Overview¶

The hdf5N2Write stage consumes single-frequency, fully accumulated N² visibility frames (one frame per frequency channel per accumulation bin, produced upstream by N2Accumulate) and repacks them into HDF5 files. Each file covers a fixed window of \(N_t\) = num_file_t consecutive time bins across all \(N_f\) science frequency channels of the configured telescope. Within a file, arrays are stored “transposed” with time as the fastest-varying (last) axis, i.e. frequency-major, to allow large contiguous writes.

Files group \(N_t\) consecutive accumulation bins by their absolute bin index:

\[\texttt{abs_file_idx} = \left\lfloor \texttt{bin_abs_index} / N_t \right\rfloor, \qquad t\text{-slot in file} = \texttt{bin_abs_index} \bmod N_t .\]

The absolute bin index is assigned upstream by N2Accumulate and is monotonic and safe across x/recv pipeline restarts. The index will reset across f-engine restarts, however. Depending on the accumulator configuration, bins are aligned either to a fixed FPGA-tick grid or to a fixed Earth-Rotation-Angle (ERA) grid (bin_in_ERA: true, with num_bins_per_rotation bins per Earth rotation and bin 0 starting at ERA = 0 just before 2000 Jan 1 noon).

Directory layout and file naming¶

At stage (receiver) startup, an acquisition directory is created under the base_dir:

<base_dir>/acq_YYYYMMDD_HHMMSS_NNNNNNNNN/

named from the UTC startup time (nanoseconds in the last field). Files being filled live in a hidden staging directory <acq>/.partial/vis_<abs_file_idx>.h5 and are renamed into the acquisition directory once finalized. In the default CHORD file mode the final name is:

vis_<abs_file_idx:010d>_<YYYYMMDDT_HHMMSS>_<NNNNNNNNN>.h5

where the timestamp (UTC, with 9-digit nanosecond suffix) is the time of the earliest FPGA tick contained in the file. (In the legacy CHIME file mode the name is instead a 9-digit count of seconds since instrument start plus a frequency suffix, NNNNNNNNN_0000.h5.)

A file is finalized (flushed, closed, renamed) when (a) all \(N_f \times N_t\) (frequency, time) slots have been filled, (b) no new frame has arrived for it for late_frame_grace_seconds while later data flows, or (c) kotekan shuts down. Frames arriving after their window was finalized are dropped. A file may therefore be partially filled: missing (frequency, time) slots are zero-filled in all datasets, /frames_added is 0 there, and frac_lost stays at its default of 1.0 (see Completeness tracking and configuration snapshots).

Dimensions, conventions, and data types¶

Dimension symbols¶

Symbol	Axis label	Meaning
\(N_f\)	`frequency`	science frequency channels (`num_file_f` attribute; the telescope’s science band)
\(N_p\)	`product`	correlation products; \(N_e(N_e+1)/2\) for the `FullUpperTri` layout
\(N_e\)	`element`	correlator inputs (typically 2 polarizations per dish)
\(N_{ev}\)	`eigenval` / `eigenvec`	eigenvalue/eigenvector pairs (`num_ev`; may be 0)
\(N_t\)	`time`	time bins per file (`num_file_t`)
\(N_d\)	`dish`	dish slots in the telescope description (`num_dishes`)
\(N_k\)	`threshold`	RFI excision threshold slots (`MAX_NUM_RFI_THRESHOLDS` = 8)
–	`pol_product`	3: polarization pairs (XX, XY, YY)
–	`xyz`	3: Cartesian components

Conventions¶

Axis labels. Every dataset created by the stage carries a string-array attribute axis naming its dimensions in order, e.g. /vis has axis = ["frequency", "product", "time"]. (Some per-input tables under /index_map carry the axis label dish for historical reasons; see below.)

Frequency axis. Index \(f\) along the frequency axis corresponds to FPGA channel freq_id \(= f_{\min} + f\), where \(f_{\min}\) is the telescope’s first science channel. The physical centre frequency and width of each channel are in /index_map/freq. The telescope may have more channels in total (num_telescope_f) than are written (num_file_f).

Product ordering (FullUpperTri). Products enumerate the upper triangle of the \(N_e \times N_e\) visibility matrix, including autocorrelations, row-major: \((0,0), (0,1), \dots, (0,N_e{-}1), (1,1), (1,2), \dots, (N_e{-}1,N_e{-}1)\). The pair for product \(p\) is stored in /index_map/prod as (input_a, input_b) with input_a <= input_b; input_a is the row and input_b the column of the matrix entry \(V_{ab}\). The lower triangle is the complex conjugate.

Element ordering. The mapping from element index to physical input (dish, polarization) is given by the input_order attribute; see Element Ordering for the defined orderings. Per-element grid positions are in the main_array_grid_indices and feed_positions_m attributes, in the same element order.

Time scales. Two time scales appear throughout: instrument time (*_t_inst_ns, also frame0_unix_ns), nanoseconds since the UNIX epoch, defined by the (ideally GPS-disciplined) FPGA clock; and UT1 (*_ut1_ns), nanoseconds since the J2000(UT1) epoch. Their difference is delta_UT1_inst (seconds). ERA is the Earth Rotation Angle in degrees \([0, 360)\); ERAL is the local ERA, \(\mathrm{ERAL} = \mathrm{ERA} + \lambda_{\mathrm{ITRS}} + s'\), with \(\lambda_{\mathrm{ITRS}}\) the telescope’s ITRS longitude. xp_as / yp_as are the polar motion components in arcseconds.

FPGA ticks. One FPGA tick (sequence number increment) is one F-engine spectrum, of duration fpga_seq_length_ns. Tick 0 occurs at instrument time frame0_unix_ns.

Complex data. Complex datasets use an HDF5 compound type {r: float32, i: float32}; h5py reads them natively as complex64.

Sentinels. Quantities that no upstream stage populated hold sentinel values rather than physical ones: erms = -1 or -FLT_MAX, gain = -1+0j, radiometer_chi2 = -1, flags = 0, and eval / evec = 0 when no eigensolver ran. Which of these are real in a given acquisition depends on the deployed pipeline.

CHIME file mode. With file_mode == CHIME (legacy), the flag-like datasets (vis_weight, flags, valid_fpga_count, rfi_fpga_count, rfi_only_fpga_count, pl_fpga_count, frac_lost, frac_rfi, frac_rfi_only, frac_pl) live under a /flags group instead of the root. Everything below assumes the default CHORD mode (all datasets at the root).

Root attributes¶

All attributes are attached to the file’s root group. Scalar HDF5 types are given as NumPy-style names.

File identity and structure¶

Attribute	Type	Description
`version`	string	File format version; `"CHORD_0.0"`.
`file_mode`	string	`"CHORD"` or `"CHIME"` (see Conventions).
`abs_file_idx`	uint64	Absolute file index (= `bin_abs_index` / \(N_t\)); also the first number in the file name.
`num_file_t`	uint64	Time bins per file, \(N_t\).
`num_file_f`	uint64	Science frequency channels in the file, \(N_f\).
`num_telescope_f`	uint64	Total telescope channels.
`num_elements`	uint32	Correlator inputs, \(N_e\).
`num_prod`	uint32	Correlation products, \(N_p\).
`num_ev`	uint32	Eigenpairs, \(N_{ev}\).
`n2_layout`	string	Visibility matrix packing: `FullUpperTri`, `RedundantBaselineAvg`, `Autocorrelations`, `InputANDMasked`, `InputORMasked`, or `GeneralSubset`.
`input_order`	string	Element ordering of the data (an `ElementOrder` name; see Element Ordering).
`num_dishes`	uint64	Dish slots in the telescope description, \(N_d\).
`digital_gains_source_file`	string	Path of the gains file copied into `/digital_gains` (only present when gains are configured).
`digital_gains_source_file_modified`	string	Only present (value `"true"`) if the gains file changed on disk between file creation and flush.

Telescope geometry and timing¶

Attribute	Type	Description
`instrument_name`	string	Telescope name, e.g. `"CHORDTelescope"`.
`nyquist_zone`	uint8	Sampling Nyquist zone.
`gps_time_enabled`	bool	True if the FPGA timestamp is GPS-disciplined (absolute times are trustworthy).
`frame0_unix_ns`	int64	Instrument time of FPGA tick 0, ns since the UNIX epoch.
`fpga_seq_length_ns`	uint64	Duration of one FPGA tick, ns.
`itrs_lat_deg`	float64	Telescope origin ITRS latitude, degrees.
`itrs_lon_deg`	float64	Telescope origin ITRS longitude, degrees.
`grid_orientation`	float64[9]	Row-major 3x3 rotation matrix from topocentric coordinates to the dish grid frame.
`dish_orientation`	float64[9]	Row-major 3x3 rotation matrix from topocentric coordinates to the dish pointing frame.
`grid_size_x`, `grid_size_y`	uint64	Dish grid dimensions.
`feed_separation_x_m`, `feed_separation_y_m`	float64	Grid spacing in metres.
`main_array_grid_indices`	int64[\(N_e\), 2]	Per-element (x, y) grid indices in element order; (-1, -1) for inputs not in the main array.
`feed_positions_m`	float64[\(N_e\), 3]	Per-element 3D feed positions in the grid frame, metres (includes per-feed displacements from the fiducial station position).
`dish_coelev_deg`	float64	Commanded dish co-elevation, degrees.

Note

Files written by older kotekan builds may lack some of these attributes or use older names: itrs_lat_deg / itrs_lon_deg were previously origin_itrs_lat_deg / origin_itrs_lon_deg, and input_order, grid_size_x, grid_size_y, feed_separation_x_m, feed_separation_y_m, main_array_grid_indices, and feed_positions_m were added later.

Earth orientation parameter (EOP) table¶

Six parallel array attributes store the EOP table that was loaded when the file was created, one entry per tabulated epoch (typically daily entries spanning the acquisition). All have the same length \(n\). They are omitted entirely if the telescope has no EOP table loaded.

Attribute	Type	Description
`EOP_t_inst_ns`	int64[n]	Epochs, instrument time (ns, UNIX epoch).
`EOP_t_ut1_ns`	int64[n]	Epochs, UT1 (ns since J2000(UT1)).
`EOP_delta_UT1_inst`	float64[n]	UT1 - instrument time, seconds.
`EOP_ERA_deg`	float64[n]	Earth Rotation Angle, degrees.
`EOP_xp_as`	float64[n]	Polar motion \(x_p\), arcseconds.
`EOP_yp_as`	float64[n]	Polar motion \(y_p\), arcseconds.

Index maps (`/index_map`)¶

Static lookup tables describing the axes. The input tables (grid_x_idx through label below) are copied verbatim from the telescope’s dish_inputs configuration, which carries one entry per correlator input (element); inputs not populated in the configuration hold type Fake (-1) and label "Fake". These tables are indexed by input even where the axis attribute says dish.

Dataset	Shape	Type	Description
`freq`	(\(N_f\))	compound {centre: f8, width: f8}	Centre frequency and channel width in MHz of each `frequency` index.
`prod`	(\(N_p\))	compound {input_a: u2, input_b: u2}	Element-index pair \((a, b)\), \(a \le b\), for each `product` index (row/column of the visibility matrix).
`grid_x_idx`	(inputs)	int64	East–west dish grid column of each input.
`grid_y_idx`	(inputs)	int64	North–south dish grid row of each input.
`dish_positions_in_grid_coords`	(\(N_d\), 3)	float64	Dish positions in metres in the grid coordinate frame (grid index x dish separation); one entry per dish.
`feed_pos_disp_m`	(inputs, 3)	float64	Feed position displacement from the nominal grid position, metres.
`coelev_disp_deg`	(inputs)	float64	Co-elevation pointing offset from the commanded `dish_coelev_deg`, degrees.
`type`	(inputs)	int32	Dish type enum: -1 = `Fake` (unpopulated input), 0 = `ArrayDish`, 1 = `RFIDish`.
`label`	(inputs)	variable-length string	Human-readable input labels; `"Fake"` for unpopulated inputs.

Visibility and per-(frequency, time) datasets¶

These carry the science data. Datasets marked [c] are written with the configured compression filter (see Chunking, compression, and reading the files); all others are uncompressed.

Dataset	Shape (axes)	Type	Description
`vis` [c]	(\(N_f\), \(N_p\), \(N_t\))	complex64	Accumulated visibilities \(V_{ab}\), normalized per valid FPGA sample (i.e. the mean correlation over the samples actually accumulated). Product ordering per `/index_map/prod`.
`vis_weight` [c]	(\(N_f\), \(N_p\), \(N_t\))	float32	Statistical weight of each visibility, an inverse-variance estimate computed by `N2Accumulate` from the scatter of sub-integrations (estimator selected by the accumulator’s `variance_mode`). 0 where the visibility is missing or invalid.
`eval` [c]	(\(N_f\), \(N_{ev}\), \(N_t\))	float32	Eigenvalues from a per-frequency eigen-decomposition of the visibility matrix. The dataset exists (with zero extent) even when \(N_{ev} = 0\).
`evec` [c]	(\(N_f\), \(N_{ev}\), \(N_e\), \(N_t\))	complex64	Corresponding eigenvectors, packed (eigenpair, element).
`erms`	(\(N_f\), \(N_t\))	float32	RMS of residual visibilities after removing the modelled eigenmodes. Sentinel (-1 or -FLT_MAX) when not computed.
`gain`	(\(N_f\), \(N_e\), \(N_t\))	complex64	Complex gain applied to each input. Sentinel -1+0j when no gain application was recorded (the F-engine digital gains are in `/digital_gains` instead).
`flags`	(\(N_f\), \(N_e\), \(N_t\))	float32	Per-input flag/weight factors from upstream flagging; 0 when unpopulated.
`radiometer_chi2`	(\(N_f\), \(N_t\), 3)	float32	Radiometer-equation consistency statistic (chi-squared) per polarization product (XX, XY, YY). Sentinel -1 when the test stage is not running.
`frames_added`	(\(N_f\), \(N_t\))	uint8	1 where an input frame was received and written into this (frequency, time) slot, 0 where the slot is zero-filled.
`valid_fpga_count`	(\(N_f\), \(N_t\))	uint64	Number of FPGA ticks actually accumulated into the bin (after RFI and packet-loss excision).
`rfi_fpga_count`	(\(N_f\), \(N_t\))	uint64	FPGA ticks excised by the RFI mask (may overlap packet loss).
`rfi_only_fpga_count`	(\(N_f\), \(N_t\))	uint64	FPGA ticks excised by RFI only (excluding ticks also lost to packet loss).
`pl_fpga_count`	(\(N_f\), \(N_t\))	uint64	FPGA ticks lost to packet loss.
`frac_lost`	(\(N_f\), \(N_t\))	float32	Fraction of the nominal bin not accumulated: 1 - `valid_fpga_count` / `frame_length_fpga_ticks`. Defaults to 1.0 in slots with no received frame.
`frac_rfi`	(\(N_f\), \(N_t\))	float32	`rfi_fpga_count` / `frame_length_fpga_ticks`.
`frac_rfi_only`	(\(N_f\), \(N_t\))	float32	`rfi_only_fpga_count` / `frame_length_fpga_ticks`.
`frac_pl`	(\(N_f\), \(N_t\))	float32	`pl_fpga_count` / `frame_length_fpga_ticks`.

Per-time-bin metadata datasets¶

All are 1-D of length \(N_t\) (axis time) unless noted. “Bin” values refer to the nominal accumulation bin on the upstream bin grid; “time-centre” values refer to the midpoint of the data actually accumulated. Identical values are shared by all frequencies of a time slot (the writer verifies consistency, within ns-level tolerance, as frames arrive).

Dataset	Type	Description
`fpga_start_tick`	uint64	FPGA sequence number of the first tick accumulated into the bin. Also determines the final file name (earliest tick in the file).
`frame_length_fpga_ticks`	uint64	Nominal accumulated length of the bin in FPGA ticks.
`bin_abs_index`	uint64	Absolute (restart-safe) accumulation bin counter (see Overview). Unfilled slots hold \(2^{64}-1\).
`time_center_t_inst_ns`	int64	Instrument time at the midpoint of the accumulated data, ns (UNIX epoch).
`time_center_ut1_ns`	int64	UT1 at the same midpoint, ns since J2000(UT1).
`bin_t_inst_ns`	int64	Instrument time at the nominal bin centre, ns (UNIX epoch).
`bin_ut1_ns`	int64	UT1 at the nominal bin centre, ns since J2000(UT1).
`bin_delta_ut1_inst`	float64	UT1 - instrument time at the bin centre, seconds.
`bin_ERA_deg`	float64	Earth Rotation Angle at the nominal bin centre, degrees \([0, 360)\).
`bin_xp_as`, `bin_yp_as`	float64	Polar motion \(x_p\), \(y_p\) at the bin centre, arcseconds.
`bin_start_ERA_deg`, `bin_end_ERA_deg`	float64	ERA at the nominal bin edges; with ERA binning these are exact multiples of the bin width.
`bin_start_ERAL_deg`, `bin_end_ERAL_deg`	float64	Local ERA (ERA + ITRS longitude + \(s'\)) at the bin edges, degrees.
`rfi_frame_excision_enabled`	bool	Whether second-stage (frame-level) RFI excision was active for this bin.
`rfi_frame_excision_num`	int32	Number of active excision thresholds (<= \(N_k\)); only the first this many entries of the two datasets below are meaningful.
`rfi_frame_excision_threshold`	float32 (\(N_t\), \(N_k\))	Spectral-kurtosis thresholds, in sigma, used for excision.
`rfi_frame_excision_fraction`	float32 (\(N_t\), \(N_k\))	Fraction of samples above the corresponding threshold required to trigger excision.

Completeness tracking and configuration snapshots¶

/frames_added: (\(N_f\), \(N_t\)), uint8. The authoritative mask of which (frequency, time) slots actually contain data. Files finalized by the grace-period or shutdown paths are partially filled; always check this mask (or equivalently frac_lost < 1) before interpreting zeros in the data datasets as measurements.
/config_json: 1-D, variable-length strings, unlimited maximum extent (chunk size 8), appended at every file flush. Each element is one JSON document recorded by kotekan’s Config Tracker: snapshots of the running kotekan configurations in the pipeline (the writer’s process and upstream senders) and of the FPGA configuration fetched from the FPGA controller. Each document carries the keys config (the configuration tree itself), json_hash, kotekan_version, kotekan_git_commit_hash, kotekan_build_branch, and kotekan_cmake_options. Empty if the tracker is disabled.

Digital gains group (`/digital_gains`)¶

Only present when gains are configured (baseband_gain_file pointing to a local gains archive, or baseband_gain_host_info naming an FPGA-controller config block to fetch it from over HTTP; the fetched copy is cached as <acq>/.partial/baseband_gains.h5). The gains archive is copied verbatim (H5Ocopy of its root group) into every output file, so its internal layout follows the pychfpga DigitalGainArchive format, not the conventions above. Typical contents:

Group attributes (from the source archive): acquisition_name, archive_version, collection_server, instrument_name, notes, system_user, type, version; plus selected_update_idx (int32), added by the writer: the index along update_time that was current for this acquisition (configuration baseband_gain_update_idx = -1 selects the latest entry).

Dataset	Shape	Type	Description
`gain_coeff`	(updates, channels, inputs)	complex64	Complex gain coefficient per update, FPGA channel (full band), and input.
`gain_exp`	(updates, inputs)	int32	Per-input binary exponent; the applied gain is `gain_coeff` x 2^gain_exp.
`compute_time`	(updates, inputs)	float64	UNIX time at which each input’s gains were computed.
`update_id`	(updates)	string	Identifier of each gain update.
`index_map/freq`	(channels)	compound {centre: f8, width: f8}	Centre/width in MHz of the FPGA channels (full band, not just the science band).
`index_map/input`	(inputs)	compound {chan_id: u2, correlator_input: S32}	Channel id and correlator input serial for each input.
`index_map/update_time`	(updates)	float64	UNIX time of each gain update.

The root attribute digital_gains_source_file records the source path.

Chunking, compression, and reading the files¶

Chunking¶

One-dimensional datasets use a single chunk equal to their full extent. Multi-dimensional datasets are chunked with each dimension capped by a per-axis block size from the stage configuration: frequency by blocksize_f, product and element by blocksize_p, time by blocksize_t; zero-length dimensions get chunk extent 1.

Compression¶

The four large datasets /vis, /vis_weight, /eval, and /evec are written with the configured compression; all other datasets are uncompressed. The options are:

compression: none (default) — no filter;
compression: deflate — zlib at compression_level (default 4);
use_bitshuffle: true — the bitshuffle HDF5 filter (filter id 32008), with compression selecting the backend codec (none, lz4, or zstd).

When bitshuffle is used, readers must have the plugin available: in Python, import hdf5plugin (or having the bitshuffle plugin on HDF5_PLUGIN_PATH) before reading those datasets with h5py. Note that h5py’s Dataset.compression property reports None for third-party filters; inspect the dataset creation property list to see them.

Minimal reading example¶

import h5py, hdf5plugin  # hdf5plugin registers the bitshuffle filter
import numpy as np

with h5py.File("vis_<idx>_<timestamp>.h5") as f:
    freq = f["index_map/freq"]["centre"]      # MHz, (Nf,)
    prod = f["index_map/prod"][:]             # (input_a, input_b)
    good = f["frames_added"][:].astype(bool)  # (freq, time) validity
    vis  = f["vis"][:]                        # (freq, prod, time) complex64
    w    = f["vis_weight"][:]
    t    = f["time_center_t_inst_ns"][:] * 1e-9   # UNIX seconds
    vis[~good[:, None, :].repeat(vis.shape[1], 1)] = np.nan

N2 visibility files (hdf5N2Write)¶