Info | ||
---|---|---|
| ||
Please log in to run requests for data products (searches). There are numerous benefits. Non-logged-in / anonymous users searches are limited to 3 months time range and the search results are only available for 3 days upon completion. For logged in users, search results are available for 21 days. Also, if we find any errors in the data that has been downloaded, we contact all affected users. It is possible that the requested searches are too big to complete within server uptime (usually two weeks). If this or any other problems occur while running your search request, ONC support will work with you to fix them. Logged-in users have access to the service help desk and help buttons. Non-logged in searches may not be re-run if interrupted by a server restart. Your browser remembers what non-logged-in searches are yours by using a local file (cookie). So if you switch browsers, computers, or clear your browser's data, you will lose your non-logged-in searches. User metadata is private. All you need to create an account is an email address. Thank you for understanding. |
Data Products
Here's the place to learn and talk about our data products - both current and future. Data products are primarily available through Data Search (Data Search Help page), as well as the Data Preview page and Hydrophone Viewer tool.
We have also developed public web services / API to access these data products, which can be used to automate data requests from Oceans 3.0. The OpenAPI page is the best way to get started with the API, while our client-side code libraries make integrating the API into your code much easier (they're available via the Quick Start documentation). Once users have developed code or software using the API, they can then deploy it in The Oceans 3.0 Sandbox so that it runs local to the data on our cloud (contact us for more info). The API will also work with any cloud and high performance computing environment such as the Digital Research Alliance of Canada facilities. Running code, making use of the API and cloud options is particularly useful for automated, repetitive analysis and for handling large data sets.
The aim of this documentation is to help our users understand and make use of our data products. It is also useful as a detailed resource about data product specifications. This documentation is maintained as data products are improved and added. User input for new and improved data products is encouraged: you can email us (info@oceannetworks.ca) or press the request support button in the top right corner anywhere in Oceans 3.0.
This documentation is organized around the dataproductid with one page for each (these pages are direct child pages as one can see to the left and in the table below). The dataproductid (used internally) represents each data product group which have a common name, theme, purpose and usually have a number of formats contained within. For example, in the image below, the "Time Series Scalar Data" is the common scalar data product for parsed sensor data, such as temperature, with numerous formats available, each format representing the exact same data, just in a different presentation. Users can find direct links to the data product child pages in the Data Search Data Product Selection page. Simply click on the links that are also the data product headings and names, doing so will open a new tab with the specific data product documentation:
Users can also find links in the Data Search cart, in the Status column:
These links appear for most but not all data products (work in progress). Users can always find the data product specific documentation linked in the table below.
How does it all work? Need a good overall reference?
Please see our Oceans 3.0 article: https://doi.org/10.3389/fmars.2022.806452 This article details the complete data acquisition process, from the deep ocean to the data products users can see, interact with and download.
New Features and Highlights
- See the latest release notes: New Features Release Notes
- ONC Science Hub news stories on Oceans 3.0: check out the latest article here
Expand | ||
---|---|---|
| ||
|
Data Products
Here's the place to learn and talk about our data products - both current and future. Data products are primarily available through Data Search (Data Search Help page), as well as the Data Preview page and Hydrophone Viewer tool.
We have also developed public web services / API to access these data products, which can be used to automate data requests from Oceans 2.0.
The aim of this documentation is to help our users understand and make use of our data products. It is also useful as a detailed resource about data product specifications. This documentation is maintained as data products are improved and added. User input for new and improved data products is encouraged: you can email us (info@oceannetworks.ca) or press the (?) button in Data Search Step 3.
New Features and Highlights
- See the latest release notes here.
- Thomson et al. detided data product for Tsunami research
- Track plots for drifter devices
- New RDI ADCP options (included in the documentation, will be available ~May 1st, 2018)
- Coming soon: improvements to Ice Buoy products, cast-delimited data products and improvements to vertical profile plots: collectively known as water-column data products.
...
width | 35% |
---|
Current Data Products
...
ID
...
Data Product
...
1
...
Time Series Scalar Data (incl. stationary and mobile scalar devices)
...
2
...
Time Series Scalar Plot (incl. stationary and mobile scalar devices)
...
3
...
Borehole Temperature Time Series Plot
...
4
...
...
5
...
...
7
...
...
9
...
...
10
...
...
14
...
...
18
...
...
19
...
...
20
...
...
21
...
...
22
...
Nortek Time Series (raw and processed formats)
...
23
...
...
24
...
ASL Acoustic Profiler Time Series (AZFP, AWCP and ZAP echosounders)
...
25
...
...
26
...
...
27
...
Satlantic Radiometer Time Series
...
30
...
...
33
...
...
34
...
...
35
...
...
38
...
...
40
...
COVIS Plume Imaging Time Series
...
41
...
COVIS Diffuse Flow Time Series
...
42
...
...
43
...
...
44
...
...
45
...
...
47
...
COVIS DIFFUSESLOW Special Run Raw Files
...
48
...
Kongsberg Mesotech Rotary Sonar Data Product - SWEEP
...
49
...
Nortek Profiler Daily Currents Plot
...
51
...
Hydrophone Spectral Probability Density (SPD) Plot
...
52
...
...
56
...
Time Series Scalar VPS Cast Data Product
...
57
...
Satlantic Radiometer VPS Cast Data Product
...
58
...
Nortek Time Series VPS Cast Data Product
...
59
...
Satlantic ISUS VPS Cast Data Product
...
60
...
ASL AWCP VPS Cast Data Product
...
61
...
Time Series Scalar Profile Plot
...
63
...
...
64
...
...
65
...
Kongsberg Mesotech Rotary Sonar Data Product - SCAN
...
66
...
Kongsberg EM Series Raw ALL Data
...
67
...
...
68
...
...
76
...
...
82
...
...
90
...
Time Distance Variable Scalar Plot
...
91
...
...
97
...
...
98
...
Kistler Accelerometer Raw Files
...
100
...
...
102
...
...
109
...
Ice Buoy Time Series Profile Plots
...
110
...
...
114
...
...
115
...
...
116
...
...
117
...
...
119
...
...
120
...
...
width | 65% |
---|
Search Type / Sort By Options
Currently, the default search type is "Instruments by Location". This is the option visible on Data Search Step 1, entitled "Sort By:". The default option, "Instruments by Location", allows user to navigate to a location and select an instrument/device category from which to request data products in Data Search Step 2. For these search requests, data from multiple instruments of the same category are combined together to form a continuous time series. For example, to form a long time series of CTD data in Saanich Inlet, an "Instruments by Location" would stitch together the data from approximately 23 different CTD device deployments. The "Variables by Location" search type option is similar, except that users navigate to a location and select a sensor category variable, such as temperature, and then temperature sensor data is stitched together into a time series from the most appropriate devices at that location; also known as a "primary sensor search" or "search by water property". For many users, this search type is advantageous, as it does not require the user to know which device categories host which sensors and which of those are the best to use. For instance, many devices have temperature sensors, often auxiliary to the devices' main function, while the CTD devices generally have the best temperature sensors. The "Instruments By Category" search type option enables users to navigate to a device category, such as Hydrophone, then to a specific device (e.g. Ocean Sonics icListen HF Hydrophone 1252 (23155)) and then to a single device deployment (e.g. Cambridge Bay (03-Sep-2013 to 16-Sep-2014)). Users of this option are generally internal users and scientists who know the exact device they are interested in.
Data Product Options
...
...
Metadata
Metadata reports available with nearly all different data products. These PDF reports are produced automatically when a data search is completed and are made available via a link adjacent to the data, see step 3 in data search help. The reports contain extensive information about the data, including instrument location, deployment, calibration, data quality and data gaps.
Mobile Data
See the mobile device page to see how data products handle data from mobile devices.
Data Availability
Data availability is indicated in step 2. of Data Search. The green data availability bar is based on archived data and may not show data for the last 24 hours (until it is archived). All data that goes through the shore-station drivers is archived in a raw format nightly: log files. Some devices provide data through FTP or HTTP file transfers; the data availability graph will be accurate in that case and data products will be available in near real-time (usually delayed by a few minutes). Although the data availability bar doesn't show it, scalar data is available live: data is usually only a few seconds delayed as it comes up the wire and through the various parsing, conversion, calibrated and QAQC steps. Many complex data products (data that is multidimensional, such as acoustic backscatter or profile data) produce data from log files. These complex data products can, since October 2015, access the raw data prior to archiving, to produce near-live data, usually delayed by a few minutes. In all, users should be able to access near real-time data for all active devices, in addition to accessing historic data from as far back as 2002 (currently, we continue to acquire historic data).
Attribution
The data products shall contain attribution information wherever feasible, so that all contributed parties maybe credited and acknowledgements given. Roles and dates will be included. The MAT file products all contain a metadata structure with citation information. While the text file products (CSV, ODV) have attribution line(s) in their headers and plots will show a shortened version of the same text (no dates) below the plot. If a special attribution is blank, the company default shall be used, if that is blank, the company name and role are used. Attribution text appearing all products may be hidden with the "is attribution required" field on the Network Console (for internal use only). All plots are capable of supporting a special logo, see the spectrogram data product as an example. The logo is taken from the highest priority citation with the "is logo required" field set to yes. Metadata reports will provide information on how to cite ONC data, alternatively, see http://www.oceannetworks.ca/data-tools/data-help/data-policy/how-cite-us.
Conventions
Time-stamps: Time-stamps are always in UTC. For file-names and string dates, the format conforms to the ISO8601 convention: yyyymmddTHHMMSS. In some cases, the millisecond portion may be added: yyyymmddTHHMMSS.FFFZ. Numerical time-stamps within data product files may follow a different format as noted on the data product pages. For instances, numeric time-stamps within MAT files are in the MATLAB serial date format. When resampling, the time-stamps are generally taken from the the centre of the resample interval.
File-names: Note that the underscore character, "_", is used to separate the components of the names, while spaces, dots and other special characters are not included in file-names. File breaks are avoided as much as possible, but do occur for many reasons, including configuration or device changes, plus some data products have daily file breaks.
...
...
...
- DEVICECODE is a descriptive string unique to each instrument.
- SENSORNAME is the sensor name as it appears in data search, and is only included if a single-sensor data product was requested.
- The first yyyymmddTHHMMSS.FFFZ is the time-stamp (ISO8601 format) of the first record in the file; the second yyyymmddTHHMMSS.FFFZ is the last time-stamp in the file (including data flagged and replaced with NaN). The date-to time stamp is not mandatory for all files; in particular, files streamed directly from the file archive will not get a data-to in their file-names. The time-stamp format optionally includes milliseconds: yyyymmddTHHMMSS.FFFZ, where 'FFF' are the milliseconds. If there is a data gap at the beginning and/or end of the search time range, the file-name dates will be different fropm the search time range. In the case of plotted products, the search time range sets the range of time axis so that users effectively control the horizontal scaling. Consequently, the file-name dates for plots will match the search time range, not the data time range.
- MODE is optional text which allows files of the same extension to be differentiated. It is used for different operation modes (Kongsberg scan or sweep for instance) or different data product options or multiple formats of the same extension. For example, scalar MAT files will get an 'ANCILLARY' when on ADCPs so they are not confused with RDI MAT files. Data product option mode strings are used on scalar data products primarily, examples: '-NaN', '-clean', '-NaN_clean_avg15minute', '-MinMax1hour', see here for more details. Other data products supply file modes as described in their documentation.
- EXT is the file extension.
For an "Instrument by Location" search (see the "Sort By:" option in Step 1), files will be named as follows: STATIONNAME_DEVICECATEGORY_SENSORNAME_yyyymmddTHHMMSS.FFFZ_yyyymmddTHHMMSS.FFFZ-MODE.EXT, where:
- STATIONNAME is the station name, including node and station names separated by dashes, for example: BarkleyCanyon-VPSUpperSlope.
- DEVICECATEGORY is the device category, such as "CTD". If there is more than one device in the category, the file will contain multiple devices combined together for a long record of data.
- SENSORNAME is the sensor name and is omitted for a device-level data product that contains multiple sensors.
- yyyymmddTHHMMSS.FFFZ, MODE and EXT are as above.
For a "Variables by Location" search (see the "Sort By:" option in Step 1), files will be named as follows: STATIONNAME_variables_SENSORCATEGORY_yyyymmddTHHMMSS.FFFZ_yyyymmddTHHMMSS.FFFZ-MODE.EXT, where:
- STATIONNAME is the station name, including node and station names separated by dashes, for example: BarkleyCanyon-VPSUpperSlope.
- SENSORCATEGORY is the sensor category, such as "Temperature" or "Conductivity". If there is more than one device in the category, the file will contain multiple devices combined together for a long record of data.
- yyyymmddTHHMMSS.FFFZ, MODE and EXT are as above.
Interoperability Partners
...
...
...
If you have any data product related questions or would like to see additional data products, please let us know.