Child pages
  • Data Gaps
Skip to end of metadata
Go to start of metadata

For time series data only:

Fill missing/bad data with NaNs (Not Number)

This option will, as it says, fill in data gaps with 'NaN' values in the data products. For CSV files, the text 'NaN' is inserted, while MAT files have a built-in type of the same name. Data gaps occur when the time difference between subsequent readings is greater than 1.9 times the sample period (otherwise known as the data rating). The NaNs are placed one sample period after the last reading before the data gaps.

This option will also keep any existing NaNs in the data. These are most often caused by the clean data option being selected, or by real NaNs being report, or when a sensor in a multi-sensor data product has no data. The metadata report accompanying the data product will elaborate on the QAQC test that was applied.

This is the default option.

Oceans 2.0 API filter: dpo_dataGaps=1

Do not fill gaps

This option will not take action to fill in data gaps.

This option will cause action to be taken to remove all NaNs in the data. The main implication of this is if the clean option had been selected, data that failed quality control tests will be removed entirely. However, there is an exception to this: for multi-sensor time series scalar data, if one sensor at a given time stamp has valid data, the entire row/time stamp cannot be removed, so the remaining sensors will be left as NaNs. For clarification, see the following example, note that QAQC flags of 1s are good data, 4s are failures and 9s are missing data:

sample time

sensor 1

sensor 1 flag

sensor 2

sensor 2 flag

Comment

12:00:00

42

1

42

1

Good row.

12:00:01

NaN

4

NaN

9

Two bad values; one QAQC failure, one data gap. If the do not fill gaps is selected, this entire row will be removed.

12:00:02

NaN

4

44

1

One good value, can't remove row.

File-name mode field

'NaN' is added to the file name when the data gaps are filled with NaNs.

Oceans 2.0 API filter: dpo_dataGaps=0

  • No labels