Page History

h2. Data Products

Here's the place to learn and talk about our data products -- both current and future.

h3. New Features and Highlights

* AddedAs CODARpart dataof products. We're currently working on merging the merge of VENUS and NEPTUNE data products into data search. We'll be adding ferry plots, ZAPwe andhave Imagenexadded productsor forimproved the May 1st release.
* Added [Nortek rawfollowing:
** [CODAR data products|DP:2246]
** forFerry allplotting Nortek devices. These are the files that would be produced by the Nortek data acquisition software for the 6 types of Nortek sonars.
* Improvements to ADCP data processing: dynamic rotation of data to external, internal or fixed attitude values, improved documentation (in the wiki and within the MAT files), added linear bin-mapping to correct for tilt and several other less significant changes.
* Added a new QAQC flag of 6 for most data products. When resampling, and down-sampling inparticular, it indicates that there was not enough data in the resample period to compute a reliable average / min/max / interpolated value, this occurs when the amount of data is less than 70% of expected. 
* *{_}NEW{_}* data products for [ASL echosounders|http://wiki.neptunecanada.ca/display/DP/24]: AZFP and AWCP devices. This is our first fully integrated data product that was developed for both VENUS and NEPTUNE networks - the data products are generated by the same code whether you request them from VENUS data download or from data search.
* Improved integration of mobile positioning and attitude data into all scalar data products, including support of ferry data products.
* See the latest release notes [here|help:New Features Release Notes].

products: [time-distance-variable scalar plots|DP:90] and [spatial scalar plots|DP:91]
** Added support for ZAP echosounders to the [ASL echosounder data products|DP:24], which already support AZFP and AWCP echosounder, the data products for which were already standardized across both networks and data portals.
** Added [Nortek raw data products|DP:22] for all Nortek devices. These are the files that would be produced by the Nortek data acquisition software for the 6 types of Nortek sonars.
** Improvements to ADCP data processing: dynamic rotation of data to external, internal or fixed attitude values, improved documentation (in the wiki and within the MAT files), added linear bin-mapping to correct for tilt and several other quality of life changes.
** Improved integration of mobile positioning and attitude data into all scalar data products, including support of ferry data products.
* Added a new QAQC flag of 6 for most data products. When resampling, and down-sampling inparticular, it indicates that there was not enough data in the resample period to compute a reliable average / min/max / interpolated value, this occurs when the amount of data is less than 70% of expected.
* See the latest release notes [here|help:New Features Release Notes].
* Coming soon: imagenex data products, state of the ocean/environment plots, bin-mapping option for ADCP data products.
{section}{column:width=47%}

h3. Current Data Products

|| ID || Ocean Networks Canada \\ || || ID \\ || Interoperability Partners \\ ||
| 1 | [Time Series Scalar Data|1] (incl. stationary and mobile scalar devices) | | 15 | [ISDM Data Product|DP:15] \\ |
| 2 | [Time Series Scalar Plot|DP:2] (incl. stationary and mobile scalar devices)\\ | | 16 \\ | [PANGAEA Data  Product|DP:16] |
| 3 | [Borehole Temperature Time Series Plot|DP:3] \\ | | 17 \\ | [POKM Data  Product|DP:17] |
| 4 | [Log File|DP:4] \\ | | | |
| 5 | [RDI ADCP Time Series|DP:5] \\ | | | |
| 7 | [Audio Data|DP:7] \\ | | | |
| 9 | [RDI ADCP Daily Current Plot|DP:9] | | | |
| 10 | [RDI ADCP Daily Intensity Plot|DP:10] | | | |
| 14 | [AVI Video|DP:14] | | | |
| 18 | [AGO Time Series  Plot|DP:18] | | | |
| 19 | [BioSonics Time Series|DP:19] | | | |
| 20 | [Satlantic ISUS Time Series|DP:20] | | | |
| 21 | [Time Series Staircase Plot|DP:21] | | | |
| 22 | [Nortek Time Series|DP:22] (raw and processed formats) \\ | | | |
| 23 \\ | [MP4 Video|DP:23]\\ | | | |
| 24 | [ASL Acoustic Profiler Time Series|DP:24] (AZFP, AWCP and ZAP echosounders) \\ | | | |
| 25 | [CSEM Receiver Time Series|DP:25] \\ | | | |
| 26 | [RDI Wave Time Series|DP:26] \\ | | | |
| 27 | [Satlantic Radiometer Time Series|DP:27] \\ | | | |
| 30 | [Imagenex Raw Data|DP:30] | | | |
| 33 | [COVIS Plume Imaging Raw Files|DP:33] | | | |
| 34 | [COVIS Diffuse Flow Raw Files|DP:33] | | | |
| 35 | [COVIS Plume Doppler Raw Files|DP:35] | | | |
| 38 \\ | [Hydrophone Array Raw Data|DP:38]\\ | | | |
| 40 \\ | [COVIS Plume Imaging Time Series|http://wiki.neptunecanada.ca/display/DP/40] \\ | | | |
| 41 \\ | [COVIS Diffuse Flow Time Series|http://wiki.neptunecanada.ca/display/DP/41] \\ | | | |
| 42 \\ | [MOV Video|DP:42]\\ | | | |
| 43 \\ | [OGG Video|DP:43]\\ | | | |
| 44 \\ | [MPG Video|DP:44]\\ | | | |
| 45 \\ | [Hydrophone Spectral Data|DP:45]\\ | | | |
| 46 \\ | [CODAR Data|DP:46]\\ | | | |
| 48 \\ | [Kongsberg Mesotech Rotary Sonar Data Product - SWEEP|http://wiki.neptunecanada.ca/display/DP/48] \\ | | | |
| 49 \\ | [Nortek Profiler Daily Currents Plot|DP:49] \\ | | | |
| 52 | [Imagenex Data Product|DP:52] | | | |
| 56 \\ | [Time Series Scalar VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/56] \\ | | | |
| 57 \\ | [Satlantic Radiometer VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/57] \\ | | | |
| 58 \\ | [Nortek Time Series VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/58] \\ | | | |
| 59 \\ | [Satlantic ISUS VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/59] \\ | | | |
| 60 \\ | [ASL AWCP VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/60] \\ | | | |
| 61 \\ | [Time Series Scalar Profile Plot|http://wiki.neptunecanada.ca/display/DP/61] \\ | | | |
| 63 \\ | [Image Set raw.zip |http://wiki.neptunecanada.ca/display/DP/63] | | | |
| 64 \\ | [Image Set bmp.zip |http://wiki.neptunecanada.ca/display/DP/64] | | | |
| 65 \\ | [Kongsberg Mesotech Rotary Sonar Data Product - SCAN|DP:65] \\ | | | |
| 66 \\ | [Kongsberg EM Series Raw ALL Data|DP:66] \\ | | | |
| 67 \\ | [COVIS Plume Doppler Data|DP:67]\\ | | | |
| 68 \\ | [EK60 Echosounder Data|DP:68]\\ | | | |
| 76 \\ | [Kongsberg EA600 Raw Data|DP:76]\\ | | | |
| 90 | [Time Distance Variable Scalar Plot|DP:90]\\ | | | |
| 91 | [Spatial Scalar Plot|DP:91]\\ | | | |
{column}\\
{column:width=50%}

h3. Data Product Options

For all scalar data products and some complex data products, users will be presented with options to customize their data products. These options are described in the individual data product pages. A compilation of the options is presented in the [data product options page|DP:Data Product Options].

h3. Data Quality

Data quality information is supplied by way of [data quality flags|DP:Quality Assurance Quality Control] and comments in the data products, as well as annotations listed in the [metadata|DP:Metadata] reports. See the [DP:Quality Assurance Quality Control] page for more information.



h3. Metadata

[Metadata|http://wiki.neptunecanada.ca/display/DP/Metadata] reports available with nearly all different data products. These PDF reports are produced automatically when a data search is completed and are made available via a link adjacent to the data, see step 3 in [data search help|http://wiki.neptunecanada.ca/display/help/Data+Search+Help]. The reports contain extensive information about the data, including instrument location, deployment, calibration, data quality and data gaps.


h3. Mobile Data

See the [mobile device page|http://wiki.neptunecanada.ca/display/DP/Positioning+and+Attitude+for+Mobile+Devices] to see how data products handle data from mobile devices.


h3. Conventions

*Time-stamps:* Time-stamps are always in UTC. For file-names and string dates, the format conforms to the  ISO8601 convention: yyyymmddTHHMMSS. In some cases, the millisecond  portion may be added: yyyymmddTHHMMSS.FFFZ. Numerical time-stamps within data product files may follow a different format as noted on the data product pages. For instances, numeric time-stamps within MAT files are in the [MATLAB serial date format|http://www.mathworks.com/help/matlab/matlab_prog/represent-date-and-times-in-MATLAB.html#bth57t1-1]. When [resampling|DP:Resampling], the time-stamps are generally taken from the the centre of the resample interval.



*File-names:* Note that the underscore character, "_", is used to separate the components of the names. It is also used to replace any spaces in the name to maintain compatibility. File breaks are avoided as much as possible, but do occur for many reasons, including configuration or device  changes, plus some data products have daily file  breaks.


For an instrument search, files are named as DEVICECODE\_{color:#000000}SENSORNAME{color}\_yyyymmddTHHMMSSZ_yyyymmddTHHMMSSZ\-{color:#000000}MODE{color}.EXT where:

* DEVICECODE is a descriptive string unique to each instrument.
* SENSORNAME is the sensor name as it appears in data search, and is only included if a single-sensor data product was requested.
* The first yyyymmddTHHMMSSZ is the time-stamp (ISO8601 format) of the first data record in the file. The second yyyymmddTHHMMSSZ is the last time-stamp of data in the file (not yet mandatory, being phased in). Notes: the time span of the data within the file may be considerably less than than the requested search time span. Instrument site-level searches are limited to the selected instrument deployment: use the 'All Available' option when selecting the time range to search for all of the data for the selected instrument and site. Note that file-name time-stamps may also have millisecond information: yyyymmddTHHMMSS.FFFZ, where 'FFF' are the milliseconds (this is being phased out).
* MODE is optional text which allows files to be differentiated. It is used for different operation modes (Kongsberg [scan|DP:65] or [sweep|DP:48] for instance) or different data product options. Data product option mode strings are only used on scalar data products, examples: '-NaN', '-clean', '-NaN_clean_avg15minute', '-MinMax1hour', see [here|DP:Data Product Options] for more details. In addition, the VPS cast products supply the cast time range and direction here.
* EXT is the file extension.

For a location search, files are named as STATIONNAME\_{color:#000000}DEVICECATEGORY{color}\_SENSORNAME_yyyymmddTHHMMSSZ_yyyymmddTHHMMSSZ\-{color:#000000}MODE{color}.EXT where
* STATIONNAME is the station name, including node and station names separated by dashes, for example: _BarkleyCanyon-VPSUpperSlope._
* {color:#000000}DEVICECATEGORY{color} is the device category, such as 'CTD'. If there is more than one device in the category, the file will contain multiple devices combined together for a long record of data.
* SENSORNAME is the sensor name and is omitted for a device-level data product that contains multiple sensors.
* yyyymmddTHHMMSSZ, MODE and EXT are as above.

h3. File formats

Additional resources for available file formats is available [here|DP:File Formats].









!data_products_mosaic.png|border=1,align=center!
{column}
{section}
If you have any data product related questions or would like to see additional data products, please [let us know|contact:Contact us].