The ONC MatLab client library contains a number of classes and functions that make access to the ONC data, through the Oceans 2.0 API, quick and easy to use.
Requires
Currently, only R2017a is supported, due to updates to the HTTP library that support payloads with non-200 status codes
Installation
The ONC client library is a Add-On Toolbox that can be installed in MatLab by performing the following
Versions
version | Date | Description | Toolbox | Source |
---|---|---|---|---|
1.0 | 6/28/2017 | Beta version | ||
1.01 | 7/18/2017 | Added getDirectScalar, getDirectRaw, getDirectFiles | onc-1.01.mltbx | onc-1.02-source.zip |
1.02 | 7/28/2017 | Resolve unhandled exception with invalid parameters. Resolved naming inconsistencies | onc-1.02.mltbx | onc-1.02-source.zip |
1.03 | 8/01/2017 | Added getDataProductUrls, downloadFile and decodeJsonFromUrl | onc-1.03.mltbx | onc-1.03-source.zip |
1.04 | 8/03/2017 | Updated production url from http://dmas.uvic.ca to http://data.oceannetworks.ca | onc-1.04.mltbx | onc-1.04-source.zip |
To download an attachment, please right click on the link and select Save Link As, Save Target As or a similar option provided by your browser. This will open a 'Save' dialog.
New versions are created to resolve known client library issues, add new functionality or handle changes to the underlying Web Services.
The latest version works with the current deployment of the underlying web services. See API Reference
Classes
class | description |
---|---|
onc | A wrapper class for access to Oceans 2.0 API web services |
ONC Class
Description
This class provides a wrapper for some for the most common Oceans 2.0 API requests, including the discovery services (locations, devices, deviceCategories, properties) and delivery (dataProductDelivery, scalardata and rawdata).
Constructor
onc()
onc(token, production, showInfo, outPath, timeout)
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
token | string | All Web Services require a token. This can be generated at http://dmas.uvic.ca/Profile. Click on the "Web Services" tab and click "Generate Token" | token=5ba514e8-1203-428c-8835-879c8173e387 |
Optional | |||
production | boolean | Indicates if the ONC Production server URL is used for service requests.
| true or false |
showInfo | boolean | Indicates if verbose script messages are displayed, such as request url and processing time information.
| true or false |
outPath | string | The path that downloaded files are saved to.
| 'c:/ONC/Download' |
timeout | int | The web service request timeout in seconds
| 30 |
Usage
o = onc('YOUR_TOKEN_HERE');
or
isProduction = true; showInfo = false; outPath = 'c:/ONC/Data'; timeout = 30; o = onc('YOUR_TOKEN_HERE',isProduction,showInfo,outPath,timeout);
Discovery methods
getLocations()
getLocations(parameters)
Returns a list of locations, as a list of struct objects.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
parameters | struct or array | A comma separated list of filter parameters, used to filter data from the ONC devices endpoint.
| struct('locationCode','BACAX') or {'locationCode','BACAX'} |
getDevices()
getDevices(parameters)
Returns a list of devices, as a list of struct objects.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
parameters | struct or array | A comma separated list of filter parameters, used to filter data from the ONC devices endpoint.
| struct('deviceCode','NORTEKADCP9917') or {'deviceCode','NORTEKADCP9917'} |
getDeviceCategories()
getDeviceCategories(parameters)
Returns a list of deviceCategories, as a list of struct objects.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
parameters | struct or array | A comma separated list of filter parameters, used to filter data from the ONC deviceCategories endpoint.
| struct('deviceCategoryCode','HYDROPHONE') or {'deviceCategoryCode','HYDROPHONE'} |
getProperties()
getProperties(parameters)
Returns a list of properties, as a list of struct objects.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
parameters | struct or array | A comma separated list of filter parameters, used to filter data from the ONC properties endpoint.
| struct('propertyCode','seawatertemperature') or {'propertyCode','seawatertemperature'} |
getDataProducts()
getDataProducts(parameters)
Returns a list of data products as a list of struct objects.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
parameters | struct or array | A dictionary of parameters, used to filter data from the ONC dataProducts endpoint.
| struct('dataProductCode','TSSD', ... 'extension','json') or {'dataProductCode','TSSD', ... 'extension','json'} |
Data Product Download methods
orderDataProduct()
orderDataProduct(parameters, maxRetries, downloadResultsOnly, includeMetadataFile)
Requests a data product and downloads the generated files to the class's outPath.
This method performs the complete data product download workflow by calling additional methods to perform the following steps:
1. Request a data product, which includes estimates for download size and time.
2. Run a requested data product, which kicks off a process on the task machine to generate the data product.
3. Download the data product to disk. The process will continue to poll the web service to determine if the product is ready to download. If it is not ready, status messages will be provided. Once it is ready, it will download the data product to disk
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
parameters | struct or array | A comma separated list of filter parameters, used to request data as a data product from the ONC dataProductDelivery endpoint.
| struct('locationCode','BACAX',... or {'locationCode','BACAX',... |
Option | |||
maxRetries | int | The number of times to retry the service before the function aborts.
| |
downloadResultsOnly | boolean | Indicates if the files will be downloaded or if only the url to the file will be returned
| |
includeMetadataFile | boolean | Indicates if the metadata file associated with the data product request will be downloaded
|
Returns
A dictionary of download results and run time statistics
url: "http://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=b6ede000-1865-4ac3-94ad-e87d8bdfd307&dpRunId=4658962&index=1" messages: {1×2 cell} status: 'complete' size: 3060227 file: 'c:\ONC\data\BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv' index: 1 downloaded: 1 url: "http://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=b6ede000-1865-4ac3-94ad-e87d8bdfd307&dpRunId=4658962&index=meta" messages: {} status: 'complete' size: 27504 file: 'c:\ONC\data\BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean_CSV_META.pdf' index: 'meta' downloaded: 1
Property | Type | Description |
---|---|---|
downloadResults | list | A cell array of download results struct objects. |
downloadResults{}.url | string | The URL used to make the download request. |
downloadResults{}.message | list | A cell array of the messages that were returned from the dataProductDelivery download method when it was polled. |
downloadResults{}.status | string | The status of the download process at the request. Possible values are:
|
downloadResults{}.size | float | The actual size of the file in bytes. |
downloadResults{}.file | string | The full path of the file that was downloaded. |
downloadResults{}.index | string | The index of the file downloaded. The index can either be a number 1-* or "meta" for the data product metadata file. |
downloadResults{}.downloaded | boolean | Indicates if the file was downloaded. If the filter downloadResultsOnly=True is used, the data product is not downloaded and the value will be False |
requestDataProduct()
requestDataProduct(parameters)
Requests a data product using the dataProductDelivery service and provides request information so that the calling function can decide if should run the data product.
The struct of information it returns includes a requestID, which is used to run the data product, and estimates of the expected download times and file sizes.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
parameters | struct or array | A comma separated list of filter parameters, used to request data as a data product from the ONC dataProductDelivery endpoint.
| struct('locationCode','BACAX',... or {'locationCode','BACAX', ... |
Returns
returns a data product request struct
for archived files (AD):
dpRequestId: 2046404 compressedFileSize: 12563408 fileSize: 70766230 numFiles: 4 downloadTimes: {1x3 cell} 10Mbps: 7.076623 50Mbps: 1.4153247 150Mbps: 0.47177488
for generated files:
dpRequestId: 2046404 estimatedFileSize: 8348187 estimatedProcessingTime: 16
Property | Type | Description | Example |
---|---|---|---|
Always contains | |||
dpRequestId | int | A unique id for a data product request |
|
May contain | |||
compressedFileSize | int | The compressed size of the known file(s) in bytes. | compressedFileSize : 12563408 |
fileSize | int | The size of known file(s) in bytes. | fileSize : 70766230 |
numFiles | int | The number of files. | numFiles : 4 |
downloadTimes | cell array | A dictionary of estimated download times | downloadTimes: {1x3 cell} 10Mbps: 7.076623 50Mbps: 1.4153247 150Mbps: 0.47177488 |
estimatedFileSize | int | The estimated file size of generated data product | estimatedFileSize: 8348187 |
estimatedProcessingTime | int | The estimated time, in seconds, that it will take to run the data product request. | estimatedProcessingTime: 16 |
runDataProduct()
runDataProduct(requestId)
Runs a data product request using the dataProductDelivery service with a Request Id and returns a list of run ids
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
requestId | int | A dataProductDelivery dpRequestId, which is returned by the dataProductDelivery service request method, or as the 'dpRequestId' property in the dpRequest returned by the requestDataProduct() method. | 2046404 |
Returns
a list of runIds
dpRunId: 4659102 status: 'data product running'
downloadDataProduct()
downloadDataProduct(runId, fileCount, estimatedProcessingTime, maxRetries, downloadResultsOnly, includeMetadataFile)
Polls the dataProductDelivery service until data product generation task is complete and downloads all of the files and/or retrieves the file information (url, file and download status).
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
runId | int | A dataProductDelivery dpRunId, which is returned by the dataProductDelivery service run method, or as the dpRunId property in the list returned by the runDataProduct() method. | 4629218 |
Option | |||
fileCount | int | The actual or estimated file count, which is returned from the dataProductDelivery request method
| 1 |
estimatedProcessingTime | int | The estimated time in seconds, that it should take to run the request on the task machine.
| 2 |
maxRetries | int | The number of times to retry the service before the function aborts.
| 10 |
downloadResultsOnly | boolean | Determines if the files will be downloaded or if only the url to the file will be returned
| true |
includeMetadataFile | boolean | Indicates if the metadata file associated with the data product request will be downloaded
| true |
Returns
returns a list of download results struct objects
url: "http://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=1" messages: {1×2 cell} status: 'complete' size: 3060227 file: 'c:\ONC\data\BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv' index: 1 downloaded: 1 url: "http://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=meta" messages: {} status: 'complete' size: 27504 file: 'c:\ONC\data\BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean_CSV_META.pdf' index: 'meta' downloaded: 1
Property | Type | Description |
---|---|---|
url | string | The URL used to make the download request. |
message | list | A list of the messages that were returned from the dataProductDelivery download method when it was polled. |
status | string | The status of the download process at the request. Possible values are:
|
size | float | The actual size of the file in bytes. |
file | string | The full path of the file that was downloaded. |
index | string | The index of the file downloaded. The index can either be a number 1-* or "meta" for the data product metadata file. |
downloaded | boolean | Indicates if the file was downloaded. If the filter downloadResultsOnly=True is used, the data product is not downloaded and the value will be False |
getDataProductUrls()
getDataProductUrls(parameters, maxRetries=100)
Orders a data product and return only the urls, for download at a later time. Urls can be used with the downloadFile() or getJsonFromUrl() methods
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
parameters | struct or array | A comma separated list of filter parameters, used to request data as a data product from the ONC dataProductDelivery endpoint.
| struct('locationCode','BACAX',... or {'locationCode','BACAX', ... |
Option | |||
maxRetries | int | The number of times to retry the service before the function aborts.
| 10 |
Returns
returns a list of urls
["http://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=1"]
downloadFile()
downloadFile(url, file, multiThreadMessages=False)
Downloads a file from a URL, write it to a file and return download results information (url, file, message and download status).
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
url | string | The url to be downloaded | http://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN_HERE> &dpRunId=<YOUR_RUN_ID_HERE>&index=<YOUR_INDEX_HERE> |
file | string | The fulll path of the file to be downloaded to. | c:/temp/myDownload.csv |
Option | |||
multiThreadMessages | boolean | Determines how the method and called method should print messages to the console. If downloading data products in a multi-threaded pattern, messages written to the console can overlap and progress dots can be written out of context
| True |
returns a downloadResults dictionary
url: "http://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=b6ede000-1865-4ac3-94ad-e87d8bdfd307&dpRunId=4722272&index=1" messages: {} file: 'c:\ONC\data\matlab\myDownload.json' downloaded: 1 status: 'complete'
Property | Type | Description |
---|---|---|
url | string | The URL used to make the download request. |
file | string | The full path of the file that was downloaded. |
message | list | A list of the messages that were returned from the dataProductDelivery download method when it was polled. |
downloaded | boolean | Indicates if the file was downloaded. |
status | string | The status of the download process at the request. Possible values are:
|
decodeJsonFromUrl()
decodeJsonFromUrl(url)
Returns a dictionary from the JSON returned from a URL
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
url | string | The url to a JSON data product delivery result | http://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN_HERE>&dpRunId=<YOUR_RUN_ID_HERE>&index=<YOUR_INDEX_HERE> |
Near Real-Time data product methods
getDirectScalar()
getDirectScalar(parameters, outputFormat, metadata, rowLimit)
Returns scalar data, in JSON Object or Array format, in the response payload.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
parameters | struct or array | A comma separated list of filter parameters, used to request data as a data product from the ONC dataProductDelivery endpoint.
| struct('locationCode','TWDP', ... 'deviceCategoryCode','TSG', ...'begin','2016-09-01T00:00:00.000Z', ... 'end','2016-09-01T08:00:00.000Z') or {'locationCode','TWDP', ... |
Option | |||
outputFormat | string | The encoding of the response output.
| "outputFormat","Array" |
metadata | string | The amount of metadata detail that is included in the response output.
| "metadata","Full" |
rowLimit | integer | The maximum number of rows of scalar data that will be returned for each sensor
| "rowLimit",5000 |
Returns
A structure that contains sensor data that matches the filter criteria defined in the input parameters, metadata and service metadata.
metadata: [1×1 struct] bbox: [1×1 struct] maxDepth: 3 maxLat: 49.1139 maxLon: -123.5264 minDepth: 3 minLat: 49.1139 minLon: -123.5264 depth: 3 deviceCategoryName: 'TSG' latitude: 49.1139 longitude: -123.5264 begin: '2016-09-01T00:00:00.000Z' end: '2016-09-01T00:00:30.000Z' locationCode: 'TWDP' sensorData: [3×1 struct] actualSamples: 1 data: [1×1 struct] qaqcFlag: 1 sampleTime: '2016-09-01T00:00:07.037Z' value: 3.4429 sensor: 'Conductivity' sensorName: 'Conductivity' unitOfMeasure: 'S/m' serviceMetadata: [1×1 struct] metadata: 'Full' nextDateFrom: '2016-09-01T00:00:17.041Z' outputFormat: 'Object' rowLimit: 100000 sensors: [] totalActualSamples: 3 begin: '2016-09-01T00:00:00.000Z' end: '2016-09-01T00:00:10.000Z' deviceCategoryCode: 'TSG' locationCode: 'TWDP'
Property | Type | Description |
---|---|---|
metadata | struct | A structure of metadata, if metadata='Full' input parameter is used |
metadata.bbox | struct | A Bounding Box structure |
metadata.bbox.maxDepth | float | The maximum depth of the instruments contributing to the results |
metadata.bbox.maxLat | float | The maximum Latitude of bounding box of the instruments contributing to the results |
metadata.bbox.maxLon | float | The maximum Longitude of the bounding box of the instruments contributing to the results |
metadata.bbox.minDepth | float | The minimum Depth of the instruments contributing to the results |
metadata.bbox.minLat | float | The minimum Latitude of bounding box of the instruments contributing to the results |
metadata.bbox.minLon | float | The minimum Longitude of the bounding box of the instruments contributing to the results |
sensorData | list | A list of sensor dictionaries |
sensorData[].acutalSamples | int | The number of samples in the data list |
sensorData[].data | list | A list of observation structures |
sensorData[].data[].qaqcFlag | int | The QAQC Flag for the observation |
sensorData[].data[].sampleTime | datetime string | The recorded time of the observation
|
sensorData[].data[].value | float | The value of the observation |
sensorData[].sensor | string | The code of the sensor that is being observed |
sensorData[].sensorName | string | The name of the sensor that is being observed |
sensorData[].unitOfMeasure | string | The unit of measure of the observation |
serviceMetadata | dictionary | A dictionary of the metadata for the service request |
serviceMetadata.metadata | string | The requested metadata filter |
serviceMetadata.nextDataFrom | datetime string | The begin date for next reading in yyyy-MM-ddTHH:mm:ss.SSSZ format |
serviceMetadata.outputFormat | string | The requested JSON output format |
serviceMetadata.rowLimit | int | The requested row limit per sensor |
serviceMetadata.sensors | string | A comma separated list of the requested sensors |
serviceMetadata.totalActualSamples | int | The number of observations expected for all of the sensors |
serviceMetadata.begin | datetime string | The begin date of the request in yyyy-MM-ddTHH:mm:ss.SSSZ format |
serviceMetadata.end | datetime string | The end date of the request in yyyy-MM-ddTHH:mm:ss.SSSZ format |
serviceMetadata.deviceCategoryCode | string | The Device Category Code of the request. See Available Device Categories for further details |
serviceMetadata.locationCode | string | The Location Code of the request. See Available Locations for further details. |
getDirectRaw()
getDirectRaw(parameters, rowLimit)
Returns raw data from an instrument in the payload, in JSON format.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
parameters | struct or cell array | A structure or array of parameters, used to request data as a data product from the ONC dataProductDelivery endpoint.
| struct('locationCode','TWDP', ... or {'locationCode','TWDP', ... |
Option | |||
rowLimit | integer | The maximum number of rows of raw data that will be returned
| "rowLimit",5000 |
Returns
A dictionary that contains sensor data that matches the filter criteria defined in the input parameters, metadata and service metadata.
data: [1×1 struct] rawData: '20160901-00:00:07.037, 12.9162, 3.44294, 28.9096' sampleTime: '2016-09-01T00:00:07.037Z' metadata: [1×1 struct] dataMetadata: [1×1 struct] firstTimestamp: '2016-09-01T00:00:07.037Z' hasMoreDataInRange: 0 lastTimestamp: '2016-09-01T00:00:07.037Z' numberOfData: 1 locationName: 'Tsawwassen - Duke Point' queryMetadata: [1×1 struct] rowLimit: 100000 begin: '2016-09-01T00:00:00.000Z' end: '2016-09-01T00:00:10.000Z' deviceCategoryCode: 'TSG' locationCode: 'TWDP'
Property | Type | Description |
---|---|---|
metadata | dictionary | A dictionary of metadata, if metadata='Full' input parameter is used |
metadata.bbox | dictionary | A Bounding Box dictionary |
metadata.bbox.maxDepth | float | The maximum depth of the instruments contributing to the results |
metadata.bbox.maxLat | float | The maximum Latitude of bounding box of the instruments contributing to the results |
metadata.bbox.maxLon | float | The maximum Longitude of the bounding box of the instruments contributing to the results |
metadata.bbox.minDepth | float | The minimum Depth of the instruments contributing to the results |
metadata.bbox.minLat | float | The minimum Latitude of bounding box of the instruments contributing to the results |
metadata.bbox.minLon | float | The minimum Longitude of the bounding box of the instruments contributing to the results |
sensorData | list | A list of sensor dictionaries |
sensorData[].acutalSamples | int | The number of samples in the data list |
sensorData[].data | list | A list of observation dictionaries |
sensorData[].data[].qaqcFlag | int | The QAQC Flag for the observation |
sensorData[].data[].sampleTime | datetime string | The recorded time of the observation
|
sensorData[].data[].value | float | The value of the observation |
sensorData[].sensor | string | The code of the sensor that is being observed |
sensorData[].sensorName | string | The name of the sensor that is being observed |
sensorData[].unitOfMeasure | string | The unit of measure of the observation |
serviceMetadata | dictionary | A dictionary of the metadata for the service request |
serviceMetadata.metadata | string | The requested metadata filter |
serviceMetadata.nextDataFrom | datetime string | The begin date for next reading in yyyy-MM-ddTHH:mm:ss.SSSZ format |
serviceMetadata.outputFormat | string | The requested JSON output format |
serviceMetadata.rowLimit | int | The requested row limit per sensor |
serviceMetadata.sensors | string | A comma separated list of the requested sensors |
serviceMetadata.totalActualSamples | int | The number of observations expected for all of the sensors |
serviceMetadata.begin | datetime string | The begin date of the request in yyyy-MM-ddTHH:mm:ss.SSSZ format |
serviceMetadata.end | datetime string | The end date of the request in yyyy-MM-ddTHH:mm:ss.SSSZ format |
serviceMetadata.deviceCategoryCode | string | The Device Category Code of the request. See Available Device Categories for further details |
serviceMetadata.locationCode | string | The Location Code of the request. See Available Locations for further details. |