The ONC Python client library contains a number of classes and functions that access the ONC data, through the Oceans 2.0 API.
Requires
2.7 or Higher
3.6 or Higher Recommended
Installation
The ONC Python client library can be installed directly from the python package repository, using the python pip module found in the Scripts folder of your python install.
$ pip install onc
If you have a previous version of the ONC Python Client Library installed, and need to get the latest version use the following:
$ pip install onc --upgrade
To show the current installed version of the onc library use the command:
$ pip show onc
If you experience privileges issues while running pip, you can try running the pip module from the python executable, using the following:
$ python -m pip install onc
Opening the command prompt with 'Run As Administrator' may be required for some Windows installs.
Additional Python package install information can be found at https://packaging.python.org/tutorials/installing-packages/
Versions
version | Date | Description | Package | Source |
---|---|---|---|---|
1.0 | 6/9/2017 | Initial version. | onc-1.0.tar.gz | |
1.1 | 6/12/2017 | Rename onc.dap.DAP class to onc.dap.ERDDAP. | onc-1.1.tar.gz | |
1.11 | 6/13/2017 | Removed unused numpy dependency. Added python-dateutil dependency to package. | onc-1.11.tar.gz | |
1.12 | 6/23/2017 | Refactored scalar and raw data access methods. Removed print methods for testing services. Renamed private functions. Added numpy references. | onc-1.12.tar.gz | onc-1.12-source.zip |
1.13 | 7/31/2017 | Renamed getJsonFromUrl() function to decodeJsonFromUrl(). | onc-1.13.tar.gz | onc-1.13-source.zip |
1.14 | 8/01/2017 | Resolved issue with getDataProductUrls method. | onc-1.14.tar.gz | onc-1.14-source.zip |
1.15 | 8/3/2017 | Updated production url from http://dmas.uvic.ca/ to http://data.oceannetworks.ca. | onc-1.15.tar.gz | onc-1.15-source.zip |
1.16 | 8/11/2017 | Rename 'parameters' parameter to 'filters' in all applicable ONC class functions | onc-1.16.tar.gz | onc-1.16-source.zip |
1.17 | 8/15/2017 | Added file parameter defult to the download() function. | onc-1.17.tar.gz | onc-1.17-source.zip |
1.18 | 8/23/2017 | ***** DO NOT USE until http://jira.neptune.uvic.ca/browse/SYS-10353 has been resolved ****** Added getDeployments() function. Updated QA url from http://qaweb2.neptune.uvic.ca to https://qa.oceannetworks.ca. Updated production url to https. | onc-1.18.tar.gz | onc-1.18-source.zip |
To download an attachment, please right click on the link and select Save Link As, Save Target As or a similar option provided by your browser. This will open a 'Save' dialog.
New versions are created to resolve known client library issues, add new functionality or handle changes to the underlying Web Services.
The latest version works with the current deployment of the underlying web services. See API Reference.
Classes
class | import | description |
---|---|---|
ONC | from onc.onc import ONC | A wrapper class for access to Oceans 2.0 API web services. |
SOS | from onc.sos import SOS | A wrapper class for access to ONC's implementation of Sensor Observation Service (SOS). |
ERDDAP | from onc.dap import ERDDAP | A wrapper class for access to datasets published on ONC's ERDDAP (OPeNDAP) server http://qadap.onc.uvic.ca/erddap/index.html. |
NERC | from onc.nerc import NERC | A wrapper class for access to controlled vocabularies stored on the NERC vocabulary server http://vocab.nerc.ac.uk/. |
ONC Class
Description
This class provides a wrapper for some for the most common Oceans 2.0 API requests, including the discovery services (locations, devices, deviceCategories, properties) and delivery (dataProductDelivery, scalardata and rawdata).
Constructor
ONC()
ONC(token, production=True, showInfo=False, outPath='c:/temp')
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
token | string | All Web Services require a token. Once logged in at https://data.oceannetworks.ca/login, your token can be retrieved or generated at https://data.oceannetworks.ca/Profile. Click on the "Web Services" tab, then click "Generate Token". | token=5ba514e8-1203-428c-8835-879c8173e387 |
Optional | |||
production | boolean | Indicates whether the ONC Production server URL is used for service requests.
| True or False |
showInfo | boolean | Indicates whether verbose script messages are displayed, such as request url and processing time information.
| True or False |
outPath | string | The path that downloaded files are saved to.
| 'c:/ONC/Download' |
Usage
from onc.onc import ONC onc = ONC('YOUR_TOKEN_HERE')
or
from onc.onc import ONC production = True showInfo = False outPath = 'c:/ONC/Data' onc = ONC('YOUR_TOKEN_HERE',production,showInfo,outPath)
Discovery methods
getLocations()
getLocations(filters={})
Returns a list of locations, as a list of dictionary objects that meet the criteria defined by the input filters.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
filters | dictionary | A dictionary of one or more filter criteria, used to return an exclusive set of location elements from the ONC locations endpoint.
| {'locationCode':'BACAX'} |
Returns
A list of location objects.
[ { "deployments": 50, "locationName": "Axis ", "depth": 984.3076, "bbox": { "maxDepth": 987, "maxLat": 48.316837, "maxLon": -126.050125, "minDepth": 981, "minLat": 48.316517, "minLon": -126.05087 }, "description": "Depth: 985 m Latitude: 48.3167 Longitude: -126.0501 Type: Stationary platform Description: Canyon axis: benthic processes, biodiversity, sediment dynamics.", "hasDeviceData": "true", "lon": -126.05033, "locationCode": "BACAX", "hasPropertyData": "true", "lat": 48.31669, "dataSearchURL": "https://data.oceannetworks.ca/DataSearch?location=BACAX" } ]
Property | Type | Description | Example |
---|---|---|---|
location[] | list | A list of location dictionaries. | |
location[].deployments | int | The number of instrument deployments that meet the filter criteria. | "deployments" : 10 |
location[].locationName | string | The full name of the location. | "locationName":"Axis (POD 1)" |
location[].depth | double | The average depth of the deployments. |
|
location[].bbox | dictionary | A Bounding Box dictionary. | |
location[].bbox.maxDepth | double | The maximum depth of the deployments. | "maxDepth" : 100 |
location[].bbox.maxLat | double | The maximum Latitude of bounding box of the deployments. | "maxLat" : 48.476740 |
location[].bbox.maxLon | double | The maximum Longitude of the bounding box of the deployments. | "maxLon" : - 123.294904 |
location[].bbox.minDepth | double | The minimum Depth of the deployments. | "minDepth" : 50 |
location[].bbox.minLat | double | The minimum Latitude of bounding box of the deployments. | "minLat" : 48.47670 |
location[].bbox.minLon | double | The minimum Longitude of the bounding box of the deployments. | "minLon" : - 123.294900 |
location[].description | string | The description of the location. | "description":"Depth: 985 m Latitude: 48.3167 Longitude: -126.0501 Type: Stationary platform Description: Canyon axis: benthic processes, biodiversity, sediment dynamics." |
location[].hasDeviceData | string | Indicates that data products can be requested using a device category for the location. | "hasDeviceData": "true" |
location[].lon | double | The average longitude of the deployments. | "lon" : - 123.294902 |
location[].locationCode | string | A code that unquely identifies a location. | "locationCode":"BACAX" |
location[].hasPropertyData | string | Indicates that data products can be requested using property code for the location. | "hasPropertyData":"true" |
location[].lat | double | The average latitude of the deployments. | "lat" : 48.47672 |
location[].dataSearchUrl | string | The location specific Data Search web page URL. | "dataSearchURL":"https://data.oceannetworks.ca/DataSearch?location=BACAX" |
getDevices()
getDevices(filters={})
Returns a list of devices, as a list of dictionary objects that meet the criteria defined by the input filters.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
filters | dictionary | A dictionary of one or more filter criteria, used to return an exclusive set of device elements from the ONC devices endpoint.
| {'deviceCode':'NORTEKADCP9917'} |
Returns
A list of device objects.
[ { "cvTerm": { "device": [ { "uri": "http://vocab.nerc.ac.uk/collection/L22/current/TOOL0888/", "vocabulary": "SeaVoX Device Catalogue" } ] }, "dataRating": [ { "dateFrom": "2008-11-01T00:00:00.000Z", "dateTo": "2010-05-27T19:27:04.000Z", "samplePeriod": 1, "sampleSize": 1 }, { "dateFrom": "2010-05-27T19:27:04.000Z", "dateTo": "None", "samplePeriod": 10, "sampleSize": 1 } ], "deviceCode": "BC_POD1_AD2M", "deviceId": 11302, "deviceLink": "https://data.oceannetworks.ca/DeviceListing?DeviceId=11302", "deviceName": "Nortek Aquadopp HR-Profiler 2965" } ]
Property | Type | Description | Example |
---|---|---|---|
device[] | list | A list of devices. | [] |
device[].cvTerm | dictionary | A dictionary of controlled vocabulary terms. | "cvTerm" : [] |
device[].cvTerm.device[] | dictionary | A list of dictionaries of controlled vocabulary term information for the device. | "cvTerm" : [ { "device" : [] } ] |
device[].cvTerm.device[].uri | string | The URI of the controlled vocabulary term. | "uri" : "http://vocab.nerc.ac.uk/collection/L22/current/TOOL0888/" |
device[].cvTerm.device[].vocabulary | string | The name of the controlled vocabulary to which the term belongs. | "vocabulary" : "SeaVoX Device Catalogue" |
device[].dataRating[] | list | A list of data ratings dictionaries. | "dataRating" : [] |
device[].dataRating[].dateFrom | datetime string | The starting datetime for a data rating.
| "dateFrom": "2008-11-01T00:00:00.000Z", |
device[].dataRating[].dateTo | datetime string | The ending datetime for a data rating.
| "dateTo": "2010-05-27T19:27:04.000Z" |
device[].dataRating[].samplePeriod | double | The sample period, in seconds, for the data rating. | "samplePeriod" : 1 |
device[].dataRating[].sampleSize | int | The size of the data sample for the data rating. | "sampleSize": 1 |
device[].deviceCode | string | A code that unquely identifes a device. | "deviceCode": "BC_POD1_AD2M" |
device[].deviceId | int | A numeric id that uniquely identifies a device. | "deviceId": 11302 |
device[].deviceLink | string | The URL link to the Devices Listing page for the specific device. | "deviceLink": "https://data.oceannetworks.ca/DeviceListing?DeviceId=11302" |
device[].deviceName | string | The name of the device. | "deviceName": "Nortek Aquadopp HR-Profiler 2965" |
getDeployments()
getDeployments(filters={})
Returns a list of deployments, as a list of dictionary objects that meet the criteria defined by the input filters.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
filters | dictionary | A dictionary of one or more filter criteria, used to return an exclusive set of device elements from the ONC devices endpoint.
| {'deviceCode':'NORTEKADCP9917'} |
Returns
A list of deployment objects.
[ { "begin": "2016-06-14T08:58:41.000Z", "depth": 982, "deviceCode": "WLFLNTU1087", "end": "None", "hasDeviceData": true, "heading": "None", "lat": 48.31658, "locationCode": "BACAX", "lon": -126.0508, "pitch": "None", "roll": "None" }, { "begin": "2016-06-14T06:05:24.000Z", "depth": 985, "deviceCode": "BC_POD1_ROTSONAR", "end": "None", "hasDeviceData": true, "heading": 90, "lat": 48.3166, "locationCode": "BACAX", "lon": -126.0508, "pitch": "None", "roll": "None" }, { "begin": "2016-06-14T06:05:24.000Z", "depth": 985, "deviceCode": "NORTEKSIGNATURE550056", "end": "None", "hasDeviceData": true, "heading": 340, "lat": 48.3166, "locationCode": "BACAX", "lon": -126.0508, "pitch": "None", "roll": "None" },... ]
Property | Type | Description | Example |
---|---|---|---|
deployment[] | [] | ||
deployment[].locationCode | string | The locationCode for the deployment location. | "locationCode":"BACAX" |
deployment[].deviceCode | string | The deviceCode for a deployed device. | "deviceCode":"SBECTD19p6813" |
deployment[].begin | datetime | The beginning datetime of the deployment. Date Time format: yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
| "begin":2010-07-27T00:00:00.000Z |
deployment[].end | datetime | The ending datetime of the deployment.
Date Time format: yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
| "end":2016-08-01T00:00:00.000Z |
deployment[].hasDeviceData | string | Indicates that data products can be requested using a device category code for the deployment. | "hasDeviceData":"true |
deployment[].depth | double | The depth of the device deployment. | "depth" :982 |
deployment[].lat | double | The latitude of the device deployment. | "lat" : 48.31658 |
deployment[].lon | double | The longitude of the device deployment. | "lon" :- 126.0508 |
deployment[].heading | double | The heading of the device deployment. | "heading" :244 |
deployment[].pitch | double | The pitch of the device deployment. | "pitch" :null |
deployment[].roll | double | The pitch of the device deployment. | "roll" :null |
getDeviceCategories()
getDeviceCategories(filters={})
Returns a list of deviceCategories, as a list of dictionary objects that meet the criteria defined by the input filters.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
filters | dictionary | A dictionary of one or more filter criteria, used to return an exclusive set of deviceCategory elements from the ONC deviceCategories endpoint.
| {'deviceCategoryCode':'HYDROPHONE'} |
Returns
A list of device category objects.
[ { "cvTerm": { "deviceCategory": [ { "uri": "http://vocab.nerc.ac.uk/collection/L05/current/130/", "vocabulary": "SeaDataNet device categories" } ] }, "description": "Conductivity Temperature (and Depth Sensor)", "deviceCategoryCode": "CTD", "deviceCategoryName": "CTD", "hasDeviceData": "true", "longDescription": " Conductivity Temperature Depth (CTD) is an abbreviated name for an instrument package that contains sensors for measuring the conductivity, temperature, and pressure of seawater. Salinity, sound velocity, and density are variables derived from sensor measurements. CTDs can carry additional instruments and sensors such as oxygen sensors, turbidity sensors and fluorometers. " } ]
Property | Type | Description | Example |
---|---|---|---|
deviceCategory[] | list | A list of device categories. | [] |
deviceCategory[].cvTerm | dictionary | A dictionary of controlled vocabulary terms. | "cvTerm": { } |
deviceCategory[].cvTerm.deviceCategory[] | dictionary | A list of dictionaries of controlled vocabulary term information for the device category. | "deviceCategory": [ { } ] |
deviceCategory[].cvTerm.deviceCategory[].uri | string | The URI of the controlled vocabulary term. | "uri": "http://vocab.nerc.ac.uk/collection/L05/current/130/" |
deviceCategory[].cvTerm.deviceCategory[].vocabulary | string | The name of the controlled vocabulary to which the term belongs. | "vocabulary": "SeaDataNet device categories" |
deviceCategory[].description | string | The short description of the device category. | "description": "Conductivity Temperature (and Depth Sensor)" |
deviceCategory[].deviceCategoryCode | string | A code to uniquely identify a device category. | "deviceCategoryCode": "CTD" |
deviceCategory[].deviceCategoryName | string | The name of the device category. | "deviceCategoryName": "CTD" |
deviceCategory[].hasDeviceData | string | Indicates that data products can be requested using the device category for a location. | "hasDeviceData": "true" |
deviceCategory[].longDescription | string | The long description of the device category. | "longDescription": " Conductivity Temperature Depth (CTD) is an abbreviated |
getProperties()
getProperties(filters={})
Returns a list of properties, as a list of dictionary objects that meet the criteria defined by the input filters.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
filters | dictionary | A dictionary of one or more filter criteria, used to return an exclusive set of property elements from the ONC properties endpoint.
| {'propertyCode':'seawatertemperature'} |
Returns
A list of property objects.
[ { "cvTerm": { "property": [], "uom": [ { "uri": "http://vocab.nerc.ac.uk/collection/P06/current/UPDB/", "vocabulary": "BODC data storage units" } ] }, "description": "Pressure", "hasDeviceData": "true", "hasPropertyData": "true", "propertyCode": "pressure", "propertyName": "Pressure", "uom": "decibar" } ]
Property | Type | Description | Example |
---|---|---|---|
property[] | list | A list of properties. | [] |
property[].cvTerm | dictionary | A dictionary of controlled vocabulary terms. | "cvTerm": { } |
property[].cvTerm.property[] | dictionary | A list of dictionaries of controlled vocabulary term information for the property. | "property": [] |
property[].cvTerm.property[].uri | string | The Uri of the controlled vocabulary term. | |
property[].cvTerm.property[].vocabulary | string | The name of the controlled vocabulary to which the term belongs. | |
property[].cvTerm.uom[] | dictionary | A list of dictionaries of controlled vocabulary term information for the unit of measure. | "uom": [] |
property[].cvTerm.uom[].uri | string | The Uri of the controlled vocabulary term. | "uri": "http://vocab.nerc.ac.uk/collection/P06/current/UPDB/" |
property[].cvTerm.uom[].vocabulary | string | The name of the controlled vocabulary to which the term belongs. | "vocabulary": "BODC data storage units" |
property[].description | string | The short description of the device category. | "description": "Pressure" |
property[].hasDeviceData | string | Indicates that data products can be requested using the property code, along with a device category for a location. | "hasDeviceData": "true" |
property[].hasPropertyData | string | Indicates that data products can be requested using the property code for a location. | "hasPropertyData": "true" |
property[].propertyCode | string | A code that uniquely identifies the property. | "propertyCode": "pressure" |
property[].propertyName | string | The name of the property. | "propertyName": "Pressure" |
property[].uom | string | The Unit of Measure used for property measurements. | "uom": "decibar" |
getDataProducts()
getDataProducts(filters={})
Returns a list of data products as a list of dictionary objects that meet the criteria defined by the input filters.
Parameter | Type | Description | Example |
---|---|---|---|
Optional | |||
filters | dictionary | A dictionary of one or more filter criteria, used to return an exclusive set of dataProduct & extension elements from the ONC dataProducts endpoint.
| {'dataProductCode':'TSSD','extension':'json'} |
Returns
A list of data product objects.
[ { "dataProductCode": "TSSD", "dataProductName": "Time Series Scalar Data", "extension": "json", "hasDeviceData": true, "hasPropertyData": true, "helpDocument": "https://wiki.oceannetworks.ca/display/DP/1" } ]
Property | Type | Description | Example |
---|---|---|---|
dataProduct[] | list | A list of data products. | [] |
dataProduct[].dataProductCode | string | A code that uniquely identifies a data product. | "dataProductCode": "TSSD" |
dataProduct[].dataProductName | string | The name of the data product. | "dataProductName": "Time Series Scalar Data" |
dataProduct[].extension | string | The file name extension for a data product. | "extension": "json" |
dataProduct[].hasDeviceData | string | Indicates that the data product can be requested using a device category code and a location code. | "hasDeviceData": true |
dataProduct[].hasPropertyData | string | Indicates that the data product can be requested using a property code and a location code or device code. | "hasPropertyData": true, |
dataProduct[].helpDocument | string | A Link URL to the Oceans 2.0 Help documentation for the specific Data Product. | "helpDocument": "https://wiki.oceannetworks.ca/display/DP/1" |
Data Product Download methods
orderDataProduct()
orderDataProduct(filters={}, maxRetries=100, downloadResultsOnly=False, includeMetadataFile=True)
Requests a data product and downloads the generated files to the class's outPath.
This method performs the complete data product download workflow by calling additional methods to perform the following steps:
1. Request a data product, which includes estimates for download size and time.
2. Run a requested data product, which kicks off a process on the task machine to generate the data product.
3. Download the data product to disk. The process will continue to poll the web service to determine if the product is ready to download. If it is not ready, status messages will be provided. Once it is ready, it will download the data product to disk.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
filters | dictionary | A dictionary of filter criteria parameters, used to request data as a data product from the ONC dataProductDelivery endpoint.
| {'locationCode':'BACAX', |
Option | |||
maxRetries | int | The number of times to retry the service before the function aborts.
| 10 |
downloadResultsOnly | boolean | Indicates whether the files will be downloaded or if only the URL to the file will be returned.
| True |
includeMetadataFile | boolean | Indicates if the metadata file associated with the data product request will be downloaded
| True |
Returns
A dictionary of download results and run time statistics.
{'downloadResults': [ { 'url': 'https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=1', 'message': [ 'Running', 'Transferring (BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv) to the FTP server' ], 'queuingTime': 0, 'runningTime': 1.567234, 'transferringTime': 59.616635, 'downloadingTime': 0.295575, 'requestCount': 40, 'status': 'complete', 'size': 3060227.0, 'file': 'c:/ONC/data/BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv', 'index': 1, 'downloaded': True}, { 'url': 'https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=meta', 'message': [], 'queuingTime': 0, 'runningTime': 0, 'transferringTime': 0, 'downloadingTime': 0.132616, 'requestCount': 1, 'status': 'complete', 'size': 27502.0, 'file': 'c:/temp/doctest/BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean_CSV_META.pdf', 'index': 1, 'downloaded': True } ], 'stats': { 'requestRequestTime': 0.328399, 'runRequestTime': 1.060785, 'queuingTime': 0, 'runningTime': 1.567234, 'transferringTime': 59.616635, 'downloadingTime': 0.428191 } }
Property | Type | Description |
---|---|---|
downloadResults | list | A list of download results dictionaries. |
downloadResults[].url | string | The URL used to make the download request. |
downloadResults[].message | list | A list of the messages that were returned from the dataProductDelivery download method when it was polled. |
downloadResults[].queuingTime | float | The time duration, in seconds, between the first and last service request, that indicated that the task was queued. |
downloadResults[].runningTime | float | The time duration, in seconds, between the first request that returned a status message that it was running and the time that the request returned that it was transferring or complete. |
downloadResults[].transferringTime | float | The time duration, in seconds, between the first request that returned a status message that it was transferring. |
downloadResults[].downloadingTime | float | The time duration, in seconds, between the start and completion of the file download. |
downloadResults[].requestCount | integer | The number of web service request that were made before the download was complete. This includes all of the requests that returned run task status messages. |
downloadResults[].status | string | The status of the download process at the request. Possible values are:
|
downloadResults[].size | float | The actual size of the file in bytes. |
downloadResults[].file | string | The full path of the file that was downloaded. |
downloadResults[].index | string | The index of the file downloaded. The index can either be a number 1-* or "meta" for the data product metadata file. |
downloadResults[].downloaded | boolean | Indicates if the file was downloaded. If the filter downloadResultsOnly=True is used, the data product is not downloaded and the value will be False |
stats | list | A summary of individual result statistics. |
stats.requestRequestTime | float | The time to complete the request request for the requested data product. |
stats.runRequestTime | float | The time to complete the run request for the requested data product. |
stats.queuingTime | float | A sum of the queueTime times from all of the run requests for the requested data product. |
stats.runningTime | float | A sum of the runningTime times from all of the run requests for the requested data product. |
stats.transferringTime | float | A sum of the transferringTime times from all of the run requests for the requested data product. |
stats.downloadingTime | float | A sum of the downloadTime times from all of the run requests for the requested data product. |
requestDataProduct()
requestDataProduct(filters={}, returnError=False)
Requests a data product using the dataProductDelivery service and provides request information so that the calling function can decide whether to run the data product.
The dictionary of information it returns includes a requestID, which is used to run the data product, and estimates of the expected download times and file sizes.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
filters | dictionary | A dictionary of filter criteria parameters, used to request data as a data product from the ONC dataProductDelivery endpoint.
| {'locationCode':'BACAX', |
Option | |||
returnError | boolean | Indicates whether error messages from the dataProductDelivery service should be returned.
| True |
Returns
A data product request dictionary.
for archived files (AD):
{ "dpRequestId":2046404, "compressedFileSize":12563408, "fileSize":70766230, "numFiles":4, "downloadTimes":{"10Mbps":7.076623,"50Mbps":1.4153247,"150Mbps":0.47177488} }
for generated files:
{ "dpRequestId":2046404, "estimatedFileSize":8348187, "estimatedProcessingTime":16 }
Property | Type | Description | Example |
---|---|---|---|
Always contains | |||
dpRequestId | int | A unique id for a data product request. |
|
May contain | |||
compressedFileSize | int | The compressed size of the known file(s) in bytes. | "compressedFileSize" :12563408 |
fileSize | int | The size of known file(s) in bytes. | "fileSize" :70766230 |
numFiles | int | The number of files. | "numFiles" :4 |
downloadTimes | dict | A dictionary of estimated download times. | "downloadTimes" :{ "10Mbps" :7.076623, "50Mbps" :1.4153247, "150Mbps" :0.47177488} |
estimatedFileSize | int | The estimated file size of the generated data product. | "estimatedFileSize":8348187 |
estimatedProcessingTime | int | The estimated time, in seconds, that it will take to run the data product request. | "estimatedProcessingTime":16 |
runDataProduct()
runDataProduct(requestId)
Runs a data product request using the dataProductDelivery service with a Request Id and returns a list of run ids
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
requestId | int | A dataProductDelivery dpRequestId, which is returned by the dataProductDelivery service request method, or as the 'dpRequestId' property in the dpRequest returned by the requestDataProduct() method. | 2046404 |
Returns
A list of runIds.
[ 4629218, 4629219 ]
downloadDataProduct()
downloadDataProduct(runId, fileCount=0, estimatedProcessingTime=1, maxRetries=100, downloadResultsOnly=False, includeMetadataFile=False, multiThreadMessages=False)
Polls the dataProductDelivery service until data product generation task is complete and downloads all of the files and/or retrieves the file information (URL, file and download status).
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
runId | int | A dataProductDelivery dpRunId, which is returned by the dataProductDelivery service run method, or as the dpRunId property in the list returned by the runDataProduct() method. | 4629218 |
Option | |||
fileCount | int | The actual or estimated file count, which is returned from the dataProductDelivery request method.
| 1 |
estimatedProcessingTime | int | The estimated time in seconds, to run the request on the task machine.
| 2 |
maxRetries | int | The number of times to retry the service before the function aborts.
| 10 |
downloadResultsOnly | boolean | Determines whether the files will be downloaded or if only the url to the file will be returned.
| True |
includeMetadataFile | boolean | Indicates if the metadata file associated with the data product request will be downloaded
| True |
multiThreadMessages | boolean | Determines how the method and called method should print messages to the console. If downloading data products in a multi-threaded pattern, messages written to the console can overlap and progress dots can be written out of context
| True |
Returns
A list of download results dictionaries.
[ { 'url': 'https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=1', 'message': [ 'Running', 'Transferring (BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv) to the FTP server' ], 'queuingTime': 0, 'runningTime': 1.567234, 'transferringTime': 59.616635, 'downloadingTime': 0.295575, 'requestCount': 40, 'status': 'complete', 'size': 3060227.0, 'file': 'c:/ONC/data/BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv', 'index': 1, 'downloaded': True}, { 'url': 'https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=meta', 'message': [], 'queuingTime': 0, 'runningTime': 0, 'transferringTime': 0, 'downloadingTime': 0.132616, 'requestCount': 1, 'status': 'complete', 'size': 27502.0, 'file': 'c:/temp/doctest/BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean_CSV_META.pdf', 'index': 'meta', 'downloaded': True } ]
Property | Type | Description |
---|---|---|
url | string | The URL used to make the download request. |
message | list | A list of the messages that were returned from the dataProductDelivery download method when it was polled. |
queuingTime | float | The time duration, in seconds, between the first and last service request, that indicated that the task was queued. |
runningTime | float | The time duration, in seconds, between the first and last service request that returned a status message that it was running and the time that the request returned that it was transferring or complete. |
transferringTime | float | The time duration, in seconds, between the first abd last service request that returned a status message that it was transferring. |
downloadingTime | float | The time duration, in seconds, between the start and completion of the file download. |
requestCount | integer | The number of web service requests that were made before the download was complete. This includes all of the requests that returned run task status messages. |
status | string | The status of the download process at the request. Possible values are:
|
size | float | The actual size of the file in bytes. |
file | string | The full path of the file that was downloaded. |
index | string | The index of the file downloaded. The index can either be a number 1-* or "meta" for the data product metadata file. |
downloaded | boolean | Indicates whether the file was downloaded. If the filter downloadResultsOnly=True is used, the data product is not downloaded and the value will be False |
getDataProdutUrls()
getDataProductUrls(filters, maxRetries=100)
Orders a data product and return only the URLs, for download at a later time. URLs can be used with the downloadFile() or getJsonFromUrl() methods.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
filters | dictionary | A dictionary of filter criteria parameters, used to request data as a data product from the ONC dataProductDelivery endpoint.
| {'locationCode':'BACAX', |
Option | |||
maxRetries | int | The number of times to retry the service before the function aborts.
| 10 |
Returns
A list of URLs
[ 'https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=1', 'https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=2' ]
downloadFile()
downloadFile(url, file, multiThreadMessages=False)
Downloads a file from a URL, write it to a file and return download results information (URL, file, message and download status).
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
url | string | The URL to be downloaded. | https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN_HERE> &dpRunId=<YOUR_RUN_ID_HERE>&index=<YOUR_INDEX_HERE> |
file | string | The full path of the file download destination.
| c:/temp/myDownload.csv |
Option | |||
multiThreadMessages | boolean | Determines how the method and called method should print messages to the console. If downloading data products in a multi-threaded pattern, messages written to the console can overlap and progress dots can be written out of context.
| True |
Returns
A downloadResults dictionary.
{ 'url': 'https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN>&dpRunId=<YOUR_RUN_ID>&index=1', 'file': 'c:/temp/myDownload.json', 'message': [], 'downloaded': True, 'downloadingTime': 0.641304 }
Property | Type | Description |
---|---|---|
url | string | The URL used to make the download request. |
file | string | The full path of the file that was downloaded. |
message | list | A list of the messages that were returned from the dataProductDelivery download method when it was polled. |
downloaded | boolean | Indicates whether the file was downloaded. |
downloadingTime | float | The time duration, in seconds, between the start and completion of the file download. |
decodeJsonFromUrl()
decodeJsonFromUrl(url)
Returns a dictionary from the JSON returned from a URL.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
url | string | The URL to a JSON data product delivery result. | https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=<YOUR_TOKEN_HERE>&dpRunId=<YOUR_RUN_ID_HERE>&index=<YOUR_INDEX_HERE> |
Returns
A structure encoded from a JSON string.
Near Real-Time data product methods
getDirectScalar()
getDirectScalar(filters={}, outputFormat='Object', metadata='Full', rowLimit=None)
Returns scalar data, in JSON Object or Array format, in the response payload that meet the criteria defined by the input filters.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
filters | dictionary | A dictionary of filter criteria parameters, used to request data and metadata from the ONC scalardata endpoint.
| { |
Option | |||
outputFormat | string | The encoding of the response output.
| "outputFormat":"Array" |
metadata | string | The amount of metadata detail included in the response output.
| "metadata":"Full" |
rowLimit | integer | The maximum number of rows of scalar data that will be returned for each sensor.
| "rowLimit":5000 |
Returns
A dictionary that contains sensor data that match the filter criteria defined in the input parameters, metadata and service metadata.
{ 'metadata': { 'bbox': { 'maxDepth': 3.0, 'maxLat': 49.080909946666665, 'maxLon': -123.37824890666667, 'minDepth': 3.0, 'minLat': 49.080909946666665, 'minLon': -123.37824890666667 }, 'depth': 3.0, 'deviceCategoryName': 'TSG', 'latitude': 49.080909946666665, 'longitude': -123.37824890666667, 'begin': '2016-09-01T00:00:00.000Z', 'end': '2016-09-01T00:00:30.000Z', 'locationCode': 'TWDP' }, 'sensorData': [ { 'actualSamples': 1, 'data': [ { 'qaqcFlag': 1, 'sampleTime': '2016-09-01T00:00:07.037Z', 'value': 3.44294 } ], 'sensor': 'Conductivity', 'sensorName': 'Conductivity', 'unitOfMeasure': 'S/m' }, { 'actualSamples': 1, 'data': [ { 'qaqcFlag': 1, 'sampleTime': '2016-09-01T00:00:07.037Z', 'value': 28.9096 } ], 'sensor': 'salinity', 'sensorName': 'Practical Salinity', 'unitOfMeasure': 'psu' }, {...} ], 'serviceMetadata': { 'metadata': 'Full', 'nextDateFrom': '2016-09-01T00:00:17.041Z', 'outputFormat': 'Object', 'rowLimit': None, 'sensors': None, 'totalActualSamples': 3, 'begin': '2016-09-01T00:00:00.000Z', 'end': '2016-09-01T00:00:10.000Z', 'deviceCategoryCode': 'TSG', 'locationCode': 'TWDP' } }
Property | Type | Description |
---|---|---|
metadata | dictionary | A dictionary of metadata, if metadata='Full' input parameter is used. |
metadata.bbox | dictionary | A Bounding Box dictionary. |
metadata.bbox.maxDepth | float | The maximum depth of the instruments contributing to the results. |
metadata.bbox.maxLat | float | The maximum Latitude of bounding box of the instruments contributing to the results. |
metadata.bbox.maxLon | float | The maximum Longitude of the bounding box of the instruments contributing to the results. |
metadata.bbox.minDepth | float | The minimum Depth of the instruments contributing to the results. |
metadata.bbox.minLat | float | The minimum Latitude of bounding box of the instruments contributing to the results. |
metadata.bbox.minLon | float | The minimum Longitude of the bounding box of the instruments contributing to the results. |
sensorData | list | A list of sensor dictionaries. |
sensorData[].acutalSamples | int | The number of samples in the data list. |
sensorData[].data | list | A list of observation dictionaries. |
sensorData[].data[].qaqcFlag | int | The QAQC Flag for the observation. |
sensorData[].data[].sampleTime | datetime string | The recorded time of the observation.
|
sensorData[].data[].value | float | The value of the observation. |
sensorData[].sensor | string | The code of the sensor that is being observed. |
sensorData[].sensorName | string | The name of the sensor that is being observed. |
sensorData[].unitOfMeasure | string | The unit of measure of the observation. |
serviceMetadata | dictionary | A dictionary of metadata for the service request. |
serviceMetadata.metadata | string | The requested metadata filter. |
serviceMetadata.nextDataFrom | datetime string | The begin date for next reading in yyyy-MM-ddTHH:mm:ss.SSSZ format. |
serviceMetadata.outputFormat | string | The requested JSON output format. |
serviceMetadata.rowLimit | int | The requested row limit per sensor. |
serviceMetadata.sensors | string | A comma separated list of the requested sensors. |
serviceMetadata.totalActualSamples | int | The number of observations expected for all specified sensors. |
serviceMetadata.begin | datetime string | The begin date of the request in yyyy-MM-ddTHH:mm:ss.SSSZ format. |
serviceMetadata.end | datetime string | The end date of the request in yyyy-MM-ddTHH:mm:ss.SSSZ format. |
serviceMetadata.deviceCategoryCode | string | The Device Category Code of the request. See Available Device Categories for further details. |
serviceMetadata.locationCode | string | The Location Code of the request. See Available Locations for further details. |
getDirectRaw()
getDirectRaw(filters={}, rowLimit=None)
Returns raw data from an instrument in the payload, in JSON format that meet the criteria defined by the input filters.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
filters | dictionary | A dictionary of filter criteria parameters, used to request data and metadata from the ONC rawdata endpoint.
| { |
Option | |||
rowLimit | integer | The maximum number of rows of raw data that will be returned.
| "rowLimit":5000 |
Returns
A dictionary that contains sensor data matching the filter criteria defined in the input parameters, metadata and service metadata.
{ 'data': [ { 'rawData': '!AIVDM,1,1,,A,14eI18gP00G<JMPL<IaCNOv62D1C,0*43', 'sampleTime': '2017-06-23T17:18:04.645Z' }, { 'rawData': '!AIVDM,1,1,,A,34eGo=P009G;qPJL:kiv3I6800li,0*5C', 'sampleTime': '2017-06-23T17:18:04.698Z' } ], 'metadata': { 'dataMetadata': { 'firstTimestamp': '2017-06-23T17:18:04.645Z', 'hasMoreDataInRange': True, 'lastTimestamp': '2017-06-23T17:18:04.698Z', 'numberOfData': 2, 'locationCode': 'Iona Shore Station' }, 'queryMetadata': { 'rowLimit': 2, 'stationCode': 'IONA', 'begin': None, 'end': None, 'deviceCategoryCode': 'AISRECEIVER' } } }
Property | Type | Description |
---|---|---|
data | dictionary | |
data[].rawData | dictionary | A raw data string representing a sample at a specific time. |
data[].sampleTime | datetime string | The recorded time of the observation.
|
metadata | dictionary | A structure containing metadata. |
metadata.dataMetadata | dictionary | A structure containing metadata about the data. |
metadata.dataMetadata.firstTimestamp | datetime string | The date time of the first record in yyyy-MM-ddTHH:mm:ss.SSSZ format. |
metadata.dataMetadata.hasMoreDataInRange | int | Whether more data are available within the range of the request. For example if 1,000 records are returned (because the rowLimit=1000), but there could be 200,000 rows as defined by the filter criteria.
|
metadata.dataMetadata.lastTimestamp | datetime string | The date time of the last record in yyyy-MM-ddTHH:mm:ss.SSSZ format. |
metadata.dataMetadata.numberOfData | int | The number of samples in the data list. |
metadata.dataMetadata.locationName | string | The full name of the location. |
metadata.queryMetadata | struct | A structure containing metadata about the data request (query). |
metadata.queryMetadata.rowLimit | int | The requested row limit per sensor. |
metadata.queryMetadata.begin | datetime string | The begin date of the request in yyyy-MM-ddTHH:mm:ss.SSSZ format. |
metadata.queryMetadata.end | datetime string | The end date of the request in yyyy-MM-ddTHH:mm:ss.SSSZ format. |
metadata.queryMetadata.deviceCategoryCode | string | The Device Category Code of the request. See Available Device Categories for further details. |
metadata.queryMetadata.locationCode | string | The Location Code of the request. See Available Locations for further details. |