Data Products
Here's the place to learn and talk about our data products - both current and future. Data products are primarily available through Data Search
(Data Search help page). We are also developing new data product APIs (web services) and a Data Preview utility that will also provide data products documented here. The aim of this documentation is to help our users understand and make use of our data products. A secondary purpose is to double as our data product specifications. This documentation is maintained as data products are improved and added. User input for new and improved data products is encouraged: comment below, press the 
button in Data Search Step 3, or [contact us].
New Features and Highlights
- As part of the merge of VENUS and NEPTUNE data products, we have added or improved many data products - see past release notes for details.
- Added device-level scalar time series plots to support State of Environment plots and Data Preview.
- Manufacturer format .RDI files are now corrected for heading/pitch/roll, including rotation of Earth co-ordinate data. Existing post-processed RDI files will need to be regenerated before RDI files will be made search-able.
- New data products for hydrophones: spectral MAT files and spectral probability density plots.
- Complex data products are now available live. These products used to be available only after daily file archiving. Affected data products include: RDI ADCP, Nortek (all), all echosounders, all Satlantic products.
- Complex data products now produce a text file 'searchStatusUpdates.txt' available in the FTP folder for each search. This file contains all of the search status updates that are displayed to users, including important information that users often miss.
- Variables by location (primary sensor) search is now available.
- See the latest release notes here.
- Coming soon: improvements to spectrograms and hydrophone products.
If you have any data product related questions or would like to see additional data products, please [let us know].Current Data Products
ID
Data Product
1
Time Series Scalar Data (incl. stationary and mobile scalar devices)
2
Time Series Scalar Plot (incl. stationary and mobile scalar devices)
3
4
5
7
9
10
14
18
19
20
21
22
Nortek Time Series (raw and processed formats)
23
24
ASL Acoustic Profiler Time Series (AZFP, AWCP and ZAP echosounders)
25
26
27
30
33
34
35
38
40
41
42
43
44
45
46
48
49
51
52
56
57
58
59
60
61
63
64
65
66
67
68
76
82
90
91
97
98
100
Data Product Options
For all scalar data products and some complex data products, users will be presented with options to customize their data products. These options are described in the individual data product pages. A compilation of the options is presented in the data product options page.
Data Quality
Data quality information is supplied by way of data quality flags and comments in the data products, as well as annotations listed in the metadata reports. See the Quality Assurance Quality Control page for more information.
Metadata
Metadata
reports available with nearly all different data products. These PDF reports are produced automatically when a data search is completed and are made available via a link adjacent to the data, see step 3 in data search help. The reports contain extensive information about the data, including instrument location, deployment, calibration, data quality and data gaps.Mobile Data
See the mobile device page
to see how data products handle data from mobile devices.Data Availability
Data availability is indicated in step 2. of Data Search. The green data availability bar is based on archived data and may not show data for the last 24 hours (until it is archived). All data that goes through the shore-station drivers is archived in a raw format nightly: log files. Some devices provide data through FTP or HTTP file transfers; the data availability graph will be accurate in that case and data products will be available in near real-time (usually delayed by a few minutes). Although the data availability bar doesn't show it, scalar data is available live: data is usually only a few seconds delayed as it comes up the wire and through the various parsing, conversion, calibrated and QAQC steps. Many complex data products (data that is multidimensional, such as acoustic backscatter or profile data) produce data from log files. These complex data products can, since October 2015, access the raw data prior to archiving, to produce near-live data, usually delayed by a few minutes. In all, users should be able to access near real-time data for all active devices, in addition to accessing historic data from as far back as 2002 (currently, we continue to acquire historic data).Conventions
Time-stamps: Time-stamps are always in UTC. For file-names and string dates, the format conforms to the ISO8601 convention: yyyymmddTHHMMSS. In some cases, the millisecond portion may be added: yyyymmddTHHMMSS.FFFZ. Numerical time-stamps within data product files may follow a different format as noted on the data product pages. For instances, numeric time-stamps within MAT files are in the MATLAB serial date format
. When [resampling], the time-stamps are generally taken from the the centre of the resample interval.File-names: Note that the underscore character, "_", is used to separate the components of the names, while spaces, dots and other special characters are not included in file-names. File breaks are avoided as much as possible, but do occur for many reasons, including configuration or device changes, plus some data products have daily file breaks.
For an instrument by category search, files are named as DEVICECODE_SENSORNAME_yyyymmddTHHMMSS.FFFZ_yyyymmddTHHMMSS.FFFZ-MODE.EXT where:
- DEVICECODE is a descriptive string unique to each instrument.
- SENSORNAME is the sensor name as it appears in data search, and is only included if a single-sensor data product was requested.
- The first yyyymmddTHHMMSS.FFFZ is the time-stamp (ISO8601 format) of the first data record in the file; the second yyyymmddTHHMMSS.FFFZ is the last time-stamp of data in the file (including data flagged and replaced with NaN). The date-to time stamp is not mandatory, as files which are streamed directly from the file archive will not get a data-to in their file-names. The time-stamp format optionally includes milliseconds: yyyymmddTHHMMSS.FFFZ, where 'FFF' are the milliseconds. The time span of the data within the file, and hence the file-name, may be considerably less than than the requested search time span. Except for many of the plotting products where the time axis is fixed to the search time range, so users can see the plot scaled as they requested. Instrument by category searches are limited to the selected instrument deployment: use the 'All Available' option when selecting the time range to search for all of the data for the selected instrument and deployment.
- MODE is optional text which allows files of the same extension to be differentiated. It is used for different operation modes (Kongsberg scan or sweep for instance) or different data product options or multiple formats of the same extension. For example, scalar MAT files will get an 'ANCILLARY' when on ADCPs so they are not confused with RDI MAT files. Data product option mode strings are used on scalar data products primarily, examples: '-NaN', '-clean', '-NaN_clean_avg15minute', '-MinMax1hour', see here for more details. Other data products supply file modes as described in their documentation.
- EXT is the file extension.
For instrument by location and variables by location search types, files are named as STATIONNAME_DEVICECATEGORY_SENSORNAME_yyyymmddTHHMMSS.FFFZ_yyyymmddTHHMMSS.FFFZ-MODE.EXT where
- STATIONNAME is the station name, including node and station names separated by dashes, for example: BarkleyCanyon-VPSUpperSlope.
- DEVICECATEGORY is the device category, such as 'CTD'. If there is more than one device in the category, the file will contain multiple devices combined together for a long record of data.
- SENSORNAME is the sensor name and is omitted for a device-level data product that contains multiple sensors.
- yyyymmddTHHMMSS.FFFZ, MODE and EXT are as above.
Interoperability Partners
These data products are linked to from Data Search Step 1.
ID
Data Product
15
16
17
File formats
Additional resources for available file formats is available here.
