h2. Data Products
Here's the place to learn and talk about our data products -- both current and future.
h3. New Features and Highlights
* Data search now incorporates the new data product option forms. It's a customizable feature for any data product and sensor or device combination. The scalar data products are now using these options. For more details see the data product options main page [here|DP:Data Product Options]. In future updates, more options will be added.
* The CSV-NaN data product has been replaced with an option under the normal CSV data product.
* Plotting data products have new data product options for clean and raw data, including the return of min/max resampling (for plots only). For more information on the functionality of the options, see [here|DP:Data Product Options].
* Instrument search has been modified to include deployed sites in the instrument tree, allowing users to quickly select the instrument and site combination that interests them. (Affects instrument searches in plotting utility, but does not affect location, network or project searches.)
* Integration of [QAQC flags|http://venus.uvic.ca/2011/05/qaqc-for-venus-scalar-data/#more-7725] in to data products is complete. Users will see some new columns in CSV data products and new fields MAT data products that contain the flags. Plot products now colour data points that fail major QAQC tests red, minor test fails are yellow. Plots also include flag icons attached to the start and end of groups of data that are tagged by QAQC. Future revisions will enable users to select various options on the data products, including filtering out data that fail QAQC checks.
* See the latest release notes [here|help:New Features Release Notes].
{section}{column:width=50%}
h3. Current Data Products
|| ID || NEPTUNE Canada \\ || || ID \\ || Interoperability Partners \\ ||
| 1 | [Time Series Scalar Data|1] | | 15 | [ISDM Data Product|DP:15] \\ |
| 2 | [Time Series Scalar Plot|DP:2] \\ | | 16 \\ | [PANGAEA Data Product|DP:16] |
| 3 | [Borehole Temperature Time Series Plot|DP:3] \\ | | 17 \\ | [POKM Data Product|DP:17] |
| 4 | [Log File|DP:4] \\ | | | |
| 5 | [RDI ADCP Time Series|DP:5] \\ | | | |
| 7 | [WAV Audio File|DP:7] \\ | | | |
| 8 | [CSV With NaNs|DP:8]\\ | | | |
| 9 | [RDI ADCP Daily Current Plot|DP:9] | | | |
| 10 | [RDI ADCP Daily Intensity Plot|DP:10] | | | |
| 11 | [RDI ADCP Daily Correlation Plot|DP:11] | | | |
| 12 | [RDI ADCP Daily Percent Good Plot|DP:12] | | | |
| 14 | [Video File|DP:14] | | | |
| 18 | [AGO Time Series Plot|DP:18] | | | |
| 19 | [BioSonics Time Series|DP:19] | | | |
| 20 | [Satlantic ISUS Time Series|DP:20] | | | |
| 21 | [Time Series Staircase Plot|DP:21] | | | |
| 22 | [Nortek Profiler Time Series|DP:22] \\ | | | |
| 24 | [ASL AWCP Time Series|DP:24] \\ | | | |
| 25 | [CSEM Receiver Time Series|DP:25] \\ | | | |
| 26 | [RDI Wave Time Series|DP:26] \\ | | | |
| 27 | [Satlantic Radiometer Time Series|DP:27] \\ | | | |
| 33 | [COVIS Plume Imaging Raw Files|DP:33] | | | |
| 34 | [COVIS Diffuse Flow Raw Files|DP:33] | | | |
| 35 | [COVIS Plume Doppler Raw Files|DP:35] | | | |
| 36 \\ | [RDI ADCP Ensemble-Averaged Time Series|DP:36] \\ | | | |
| 40 \\ | [COVIS Plume Imaging Time Series|http://wiki.neptunecanada.ca/display/DP/40] \\ | | | |
| 41 \\ | [COVIS Diffuse Flow Time Series|http://wiki.neptunecanada.ca/display/DP/41] \\ | | | |
| 48 \\ | [Kongsberg Mesotech Rotary Sonar Data Product - SWEEP|http://wiki.neptunecanada.ca/display/DP/48] \\ | | | |
| 49 \\ | [Nortek Profiler Daily Currents Plot|DP:49] \\ | | | |
| 52 | [Imagenex Data Product|DP:52] | | | |
| 56 \\ | [Time Series Scalar VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/56] \\ | | | |
| 57 \\ | [Satlantic Radiometer VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/57] \\ | | | |
| 58 \\ | [Nortek Time Series VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/58] \\ | | | |
| 59 \\ | [Satlantic ISUS VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/59] \\ | | | |
| 60 \\ | [ASL AWCP VPS Cast Data Product|http://wiki.neptunecanada.ca/display/DP/60] \\ | | | |
| 61 \\ | [Time Series Scalar Profile Plot|http://wiki.neptunecanada.ca/display/DP/61] \\ | | | |
| 62 \\ | [Nortek ADCP Ensemble-Averaged Time Series|http://wiki.neptunecanada.ca/display/DP/62]\\ | | | |
| 63 \\ | [Image Set raw.zip |http://wiki.neptunecanada.ca/display/DP/63] | | | |
| 64 \\ | [Image Set bmp.zip |http://wiki.neptunecanada.ca/display/DP/64] | | | |
| 65 \\ | [Kongsberg Mesotech Rotary Sonar Data Product - SCAN|DP:65] \\ | | | |
{column}\\
{column:width=50%}
h3. Metadata
[Metadata|http://wiki.neptunecanada.ca/display/DP/Metadata] reports are now available with many different data products, including the scalar data products. These PDF reports are produced automatically when a data search is completed and are made available via a link adjacent to the data, see step 3 in [data search help|http://wiki.neptunecanada.ca/display/help/Data+Search+Help]. The reports contain extensive information about the data, including instrument location, deployment, calibration, data quality and data gaps.
h3. Conventions
*Time-stamps:* Time-stamps are always in UTC. For file-names and string dates, the format conforms to the ISO8601 convention: yyyymmddTHHMMSS. In some cases, the millisecond portion may be added: yyyymmddTHHMMSS.FFFZ. Numerical time-stamps within data product files may follow a different format as noted on the data product pages. For instances, numeric time-stamps within MAT files are in the [MATLAB serial date format|http://www.mathworks.com/help/matlab/matlab_prog/represent-date-and-times-in-MATLAB.html#bth57t1-1]. When [resampling|DP:Resampling], the time-stamps are generally taken from the the centre of the resample interval.
*File-names:* Note that the underscore character, "_", is used to separate the components of the names. It is also used to replace any spaces in the name to maintain compatibility. File breaks occur for many reasons, including configuration or device changes (location search only), plus some data products have daily file breaks.
For an instrument search, files are named as DEVICECODE\_{color:#000000}SENSORNAME{color}\_yyyymmddTHHMMSSZ_yyyymmddTHHMMSSZ\-{color:#000000}MODE{color}.EXT where:
* DEVICECODE is a descriptive string unique to each instrument.
* SENSORNAME is the sensor name as it appears in data search, and is only included if a single-sensor data product was requested.
* The first yyyymmddTHHMMSSZ is the time-stamp (ISO8601 format) of the first data record in the file. The second yyyymmddTHHMMSSZ is the last time-stamp of data in the file. If the data is continuous, this second time-stamp can be redundant, therefore it may be omitted for some data products (this behaviour will be phased out). The time span of the data within the file may be considerably less than than the requested search time span. Instrument site-level searches are limited to the selected instrument deployment: use the 'All Available' option when selecting the time range to search for all of the data for the selected instrument and site. Note that file-name time-stamps may also have millisecond information: yyyymmddTHHMMSS.FFFZ, where 'FFF' are the milliseconds (this feature will be phased out).
* MODE is optional text which allows files to be differentiated. It is used for different operation modes (Kongsberg [scan|DP:65] or [sweep|DP:48] for instance) or different [data product options|DP:Data Product Options], such as \-NaN, \-NaN_clean, \-clean which are used for time series scalar data. In addition, the VPS cast products supply the cast time range and direction here.
* EXT is the file extension.
For a location search, files are named as STATIONNAME\_{color:#000000}DEVICECATEGORY{color}\_SENSORNAME_yyyymmddTHHMMSSZ_yyyymmddTHHMMSSZ\-{color:#000000}MODE{color}.EXT where
* STATIONNAME is the station name, including node and station names separated by dashes, for example: _BarkleyCanyon-VPSUpperSlope._
* {color:#000000}DEVICECATEGORY{color} is the device category, such as 'CTD'. If there is more than one device in the category, the file will contain multiple devices combined together for a long record of data.
* SENSORNAME is the sensor name and is omitted for a device-level data product that contains multiple sensors.
* yyyymmddTHHMMSSZ, MODE and EXT are as above.
h3. File formats
Additional resources for available file formats is available [here|DP:File Formats].
!data_products_mosaic.png|border=1,align=center!
{column}
{section}
If you have any data product related questions or would like to see additional data products, please [let us know|contact:Contact us].
{recently-updated}
----
{portal:USER_PORTAL}
h3. |