MVF version 1 (Fringe Finder)¶

Last updated: 9 March 2010

This documents the HDF5 file format as written by augment version 4. Also indicated by (**) are those parts of the data file used by the scape package, which handles single-dish- or single baseline back-end processing for the Fringe Finder. HDF5 files contain a hierarchical structure inside, with groups connected as nodes in a directed graph. Each group may contain datasets (multi-dimensional arrays) and attributes (metadata such as strings and smaller arrays). More information on these concepts can be found on the HDF5 website. The format currently calls for a single HDF5 file per experiment, but this may change in future.

The hierarchical structure of a typical data set, with levels ‘Experiment’ -> ‘Compound Scan’ -> ‘Scan’, is reflected in the HDF5 group structure. The root ‘/’ group of the file represents the ‘Experiment’ level and contains the groups ‘Antennas’, ‘Correlator’, ‘Scans’. In addition, the root ‘/’ group the augment log dataset.

The ‘Antennas’ group contains a group for each physical antenna, named ‘Antenna%d’, where %d should be replaced by the one-based physical antenna number.

The ‘Scans’ group contains a group for each compound scan, named ‘CompoundScan%d’, where %d should be replaced by the zero-based compound scan index integer. Each ‘CompoundScan%d’ group contains a group for each scan, named ‘Scan%d’, where %d should be replaced by the zero-based scan index integer. The group structure of a typical dataset is shown below:

/
/Antennas
/Antennas/Antenna1
/Antennas/Antenna1/H
/Antennas/Antenna1/V
/Antennas/Antenna1/Sensors
/Antennas/Antenna2
/Antennas/Antenna2/H
/Antennas/Antenna2/V
/Antennas/Antenna2/Sensors
/Correlator
/Scans
/Scans/CompoundScan0
/Scans/CompoundScan0/Scan0
/Scans/CompoundScan0/Scan1
/Scans/CompoundScan0/Scan2
/Scans/CompoundScan1
/Scans/CompoundScan1/Scan0
/Scans/CompoundScan1/Scan1
/Scans/CompoundScan1/Scan2

The following sections list the data entries in each group. Four fields describe each entry: its name, D or A for whether it is an HDF5 dataset or attribute, and its data type, separated by colons and followed by a general description on the next line.

The root ‘/’ group¶

_ experiment_id : A : string

The ID for this experiment (usually a UUID generated by the system at observe time).

_ observer : A : string

The observer who ran the experiment. Data files are copied into the archive using the observer as part of the file hierarchy.

_ description : A : string

Description of the experiment as specified by the observer at observe time.

_ k7w_file_version : A : integer, optional

Version of the k7writer program used to generate the raw data file

_ data_unit : A : string, one of {‘counts’, ‘K’, ‘Jy’}

Power unit recorded in the scans, typically ‘counts’ for uncalibrated correlator output or ‘K’ for Tsys-corrected data.

data_timestamps_at_sample_centers : A : bool

Boolean flag indicating whether data timestamps are aligned with the center (if True or 1) or the start (if False or 0) of each sample / integration period.
augment_version : A : string, optional

String added by augmenter, indicating the version of the augmented data format.
augment : A : string

String added by augmenter, indicating when the file was augmented. If this attribute is absent, the file contains unaugmented correlator data only and cannot be loaded by scape.

_ augment_log : D : record array, optional, of shape (N,2) with record:

{'section' : string, 'message' : string}

The augment program log ouput from the augmentation of this HDF5 file.

/Antennas¶

This has no additional attributes or datasets.

/Antennas/Antenna%d¶

_ description : A : string

Description string of antenna, used by katpoint package to construct katpoint.Antenna object via katpoint.construct_antenna(). The string includes antenna name, location, size, etc.

/Antennas/Antenna%d/H¶

_ dbe_input : A : string

DBE input mapping for the H channel.

_ delay_s : A : float64 (???)

Cable delay in seconds for the H channel.

_ coupler_nd_model : D : float64 array of shape (N,2) ???

Table containing N frequencies in Hz in the first column and measured temperatures in K in the second column

_ pin_nd_model : D : float64 array of shape (N,2) ???

Table containing N frequencies in Hz in the first column and measured temperatures in K in the second column

/Antennas/Antenna%d/V¶

_ dbe_input : A : string

DBE input mapping for the V channel.

_ delay_s : A : float64 (???)

Cable delay in seconds for the V channel.

_ coupler_nd_model : D : float64 array of shape (N,2) ???

Table containing N frequencies in Hz in the first column and measured temperatures in K in the second column

_ pin_nd_model : D : float64 array of shape (N,2) ???

Table containing N frequencies in Hz in the first column and measured temperatures in K in the second column

/Antennas/Antenna%d/Sensors¶

This is comprised of a set of record arrays, each corresponding to sensors on the telescope. Each record array has 4 attributes: name, description, units, type (??? indicate here which items are optional for scape):

_ name : A : string

The katcp name of the sensor.

_ description : A : string, optional

Human understandable (hopefully) description of sensor.

_ units : A : string, optional

The units for the sensor readings. Currently scape does not take any notice of these attributes and assumes ‘standard’ units for the sensors.

_ type : A : string, optional???

Data type for the sensor readings.

The set of record arrays (all optional for scape ???) is as below and may be missing if no sensor data was recorded during the collection of the correlator data (???). The ‘status’ field for each is the sensor status recorded along with the sensor data. This could be ‘nominal’, ‘warn’ (sensor in warning range), ‘error’ (sensor in error range), ‘failure’ (comms error to sensor), ‘unknown’ (no initial value yet). Currently, scape does not take any notice of these sensor statuses (???).

enviro_air_pressure : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Environmental measurements of air pressure at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The air pressure is assumed by scape to be in mbar. The status is the sensor status as described above.
enviro_air_relative_humidity : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Environmental measurements of air relative humidity at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The humidity is assumed by scape to be a percentage. The status is the sensor status as described above.
enviro_air_temperature : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Environmental measurements of air temperature at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The air temperature is assumed by scape to be in deg C. The status is the sensor status as described above.
enviro_wind_direction : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Environmental measurements of wind direction at arbitrary time instants, where N is the number of wind data records. The timestamps are UTC seconds since the Unix epoch. The wind direction is assumed by scape to be in degrees increasing clockwise from North. The status is the sensor status as described above.
enviro_wind_speed : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Environmental measurements of wind speed at arbitrary time instants, where N is the number of wind data records. The timestamps are UTC seconds since the Unix epoch. The wind speed is assumed by scape to be in metres per second. The status is the sensor status as described above.
pos_actual_pointm_azim : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Measurements of actual azimuth after pointing model at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The azimuth values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_actual_pointm_elev : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Measurements of actual elevation after pointing model at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The elevation values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_actual_refrac_azim : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Measurements of actual azimuth after refraction correction at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The azimuth values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_actual_refrac_elev : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Measurements of actual elevation after refraction correction at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The elevation values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_actual_scan_azim : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Measurements of actual azimuth after scan offset at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The azimuth values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_actual_scan_elev : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Measurements of actual elevation after scan offset at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The elevation values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_request_pointm_azim : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Requested azimuth after pointing model at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The azimuth values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_request_pointm_elev : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Requested elevation after pointing model at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The elevation values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_request_refrac_azim : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Requested azimuth after refraction correction at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The azimuth values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_request_refrac_elev : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Requested elevation after refraction correction at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The elevation values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_request_scan_azim : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Requested azimuth after scan offset at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The azimuth values is assumed by scape to be in deg. The status is the sensor status as described above.
pos_request_scan_elev : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : float32, 'status' : string}
```
Requested elevation after scan offset at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The elevation values is assumed by scape to be in deg. The status is the sensor status as described above.
rfe3_rfe15_noise_coupler_on : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : string, 'status' : string}
```
Coupler noise diode firing flag at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The values are boolean wrapped in strings (‘0’ for off or ‘1’ for on). The status is the sensor status as described above.
rfe3_rfe15_noise_pin_on : D : record array of shape (N,3), with record:
```
{'timestamp' : float64, 'value' : string, 'status' : string}
```
Pin noise diode firing flag at arbitrary time instants, where N is the number of data records. The timestamps are UTC seconds since the Unix epoch. The values are boolean wrapped in strings (‘0’ for off or ‘1’ for on). The status is the sensor status as described above.

/Correlator¶

_ instrument_type : A : integer, optional

Correlator instrument type as per DBE.

_ instance_id : A : integer, optional

Correlator instance id as per DBE.

_ channel_bandwidth_hz : A : uint64, optional???

The width of each frequency channel in the scan data in Hz.

_ adc_sample_rate : A : uint64, optional???

ADC sample rate in Hz.

_ accum_per_int : A : uint64, optional???

Number of FFT frames (polyphase filterbank output samples) per integration. This is the number of samples from the output of each PFB channel that are multiplied together and accumulated in the correlator to form a single visibility sample.

_ num_freq_channels : A : uint64, optional???

Number of frequency channels contained in the scan data.

_ dump_rate_hz : A : float64

Correlator dump rate, in Hz. This should satisfy:
dump_rate = adc_sample_rate / (2 * num_freq_channels * accum_per_int)

_ center_frequency_hz : A : float32???, optional

Center frequency of the scan data in Hz.

_ channel_select : D : bool array of shape (F)

Array of boolean values, indicating which channels should be processed. F is the number of frequency channels.

_ input_map : D : record array of shape (N,2) with record:

 {'correlator_product_id' : integer, 'dbe_inputs' : string}

This is used, combined with the /Antennas/Antenna%d/H/dbe_input or
/Antennas/Antenna%d/V/dbe_input attribute to map the physical antenna
H or V channel to the DBE input used then through to the correlator output.
*N* is the number of correlation products.

/Scans¶

This has no additional attributes or datasets.

/Scans/CompoundScan%d¶

_ label : A : string

A label for this compound scan.

_ target : A : string

Description string of target, used by katpoint package to construct katpoint.Target object via katpoint.construct_target().

pointing_model : D : float32 array of shape (22,)

Pointing model used during experiment.

CorrelatorConfig¶

center_freqs : D : float64 array of shape (F,), optional

Center frequency of each channel, in Hz, where F is the number of channels. This is the main specification for center frequencies. If this dataset is not present, the center frequencies are assumed to be regularly spaced and calculated from the center_frequency_hz, bandwidth_hz and num_freq_channels attributes of this group, which must be present in this case.
bandwidths : D : float64 array of shape (F,), optional

Bandwidth of each channel, in Hz, where F is the number of channels. This is the main specification for channel bandwidths. If this dataset is not present, the bandwidths are all set to the bandwidth_hz attribute, which must be present in this case.
center_frequency_hz : A : uint64, optional

Center frequency of entire spectral band encompassing all channels. This is used to calculate channel center frequencies in the absence of the center_freqs dataset.
channel_bandwidth_hz : A : uint64, optional

Bandwidth of each channel. This is used as channel bandwidths in the absence of the bandwidths dataset.
num_freq_channels : A : uint64, optional

Number of channels.
adc_sample_rate : A : uint64, optional

ADC sample rate, in Hz.

Scan%d¶

data : D : record array of shape (T, F), with record:
```
{'AxBx' : {'r' : float32, 'i' : float32}, 'AyBy' : {'r' : float32, 'i' : float32},
 'AxBy' : {'r' : float32, 'i' : float32}, 'AyBx' : {'r' : float32, 'i' : float32}}
```
The record structure is compatible with the NumPy dtype:
```
[('AxBx', complex64), ('AyBy', complex64), ('AxBy', complex64), ('AyBx', complex64)]
```
Correlation data for a single baseline between antennas A and B (specified by antenna and antenna2, respectively), involving the x and y DBE inputs for each antenna (typically mapping to the H and V feeds, respectively). If antennas A and B correspond to the DBE antennas 0 and 1, respectively, the term ‘AxBy’ would refer to the correlation between the 0x and 1y DBE input signals. For a single dish, antennas A and B refer to the same antenna. Each correlation datum consists of a real (‘r’) and imaginary (‘i’) part, specified as 32-bit floats. The array has T rows and F columns, where T is the number of time samples in the scan and F is the number of frequency channels in the data set. The data has been scaled by accum_per_int, the number of samples that have been integrated into a single visibility.
timestamps : D : uint64 or float64 array of shape (T,)

Timestamps of the start of each sample in number of UTC milliseconds since the Unix epoch, which is the native format of the DBE. T is the number of time samples in the scan. If data_timestamps_at_sample_centers is True, the timestamps are aligned with the middle of each sample period instead.
pointing : D : record array of shape (T,), optional, with record:
```
{'az' : float32, 'el' : float32}
```
Pointing information, consisting of azimuth and elevation values per time sample, where T is the number of time samples in the scan. All angles are in degrees. The elevation should be between -90 and 90 degrees, while azimuth has no restrictions, but is nominally between -180 and 180 degrees. If this dataset is present, the original pointing information has been processed by selecting a specific sensor and interpolating its measured values to coincide with the correlator data timestamps (timestamps). For interferometric data, this is the pointing data of the first antenna.
requested_pointing : D : record array of shape (Tp,), optional, with record:
```
{'timestamp' : float64,
 'request_scan_azim' : float32, 'request_scan_elev' : float32,
 'request_refrac_azim' : float32, 'request_refrac_elev' : float32,
 'request_pointm_azim' : float32, 'request_pointm_elev' : float32}
```
The requested / predicted / commanded target position at various points in the coordinate conversion chain, at arbitrary time instants. The timestamps are UTC seconds since the Unix epoch. The rest of the field names are the names of the corresponding Fringe Finder sensors. These fields contain azimuth and elevation values per time sample, where Tp is the number of pointing data records. All angles are in degrees. The elevation should be between -90 and 90 degrees, while azimuth has no restrictions, but is nominally between -180 and 180 degrees. The scan fields represent the highest level, obtained at the input of the refraction correction step, while the refrac fields are obtained at the output of this step. The pointm fields represent the lowest level, obtained at the output of the pointing model correction step. This dataset may be missing if no new pointing info was recorded during the collection of the correlator data. For interferometric data, this is the pointing data of the first antenna.
actual_pointing : D : record array of shape (Tp,), optional, with record:
```
{'timestamp' : float64,
 'actual_scan_azim' : float32, 'actual_scan_elev' : float32,
 'actual_refrac_azim' : float32, 'actual_refrac_elev' : float32,
 'actual_pointm_azim' : float32, 'actual_pointm_elev' : float32}
```
The actual / measured target position at various points in the coordinate conversion chain, at arbitrary time instants. The timestamps are UTC seconds since the Unix epoch. The rest of the field names are the names of the corresponding Fringe Finder sensors. These fields contain azimuth and elevation values per time sample, where Tp is the number of pointing data records. All angles are in degrees. The elevation should be between -90 and 90 degrees, while azimuth has no restrictions, but is nominally between -180 and 180 degrees. The pointm fields represent the lowest level, obtained at the input of the reverse pointing model correction step, while the refrac fields are obtained at the output of this step. The scan fields represent the highest level, obtained at the output of the reverse refraction correction step. This dataset may be missing if no new pointing info was recorded during the collection of the correlator data. For interferometric data, this is the pointing data of the first antenna.
flags : D : record array of shape (T,), with record:
```
{'valid' : bool, 'nd_on' : bool}
```
Flags per time sample, where T is the number of time samples in the scan.
enviro_ambient : D : record array of shape (Ta,), optional, with record:
```
{'timestamp' : float64,
 'temperature' : float32, 'pressure' : float32, 'humidity' : float32}
```
Slowly-varying (‘ambient’) environmental measurements at arbitrary time instants, where Ta is the number of ambient environment data records. The timestamps are UTC seconds since the Unix epoch. The ambient temperature is in degrees Celsius, the atmospheric pressure is in hPa and the relative humidity is a percentage. This dataset may be missing if no new ambient environment info was recorded during the collection of the correlator data.
enviro_wind : D : record array of shape (Tw,), optional, with record:
```
{'timestamp' : float64, 'wind_speed' : float32, 'wind_direction' : float32}
```
Environmental measurements of wind speed and direction (which typically vary faster than the ambient data) at arbitrary time instants, where Tw is the number of wind data records. The timestamps are UTC seconds since the Unix epoch. The wind speed is in metres per second, and the wind direction is in degrees increasing clockwise from North. This dataset may be missing if no new wind environment info was recorded during the collection of the correlator data.
label : A : string

String that can be used to identify the type of scan in the back-end scripts. Typical contents are ‘scan’ to indicate a normal scan, ‘slew’ to indicate the telescope moving to the start of the next scan, and ‘cal’ to indicate a noise diode firing. These suggestions are not enforced or checked by the scape package, however.
comment : A : string

Generic comment added to scan.