Page tree
Skip to end of metadata
Go to start of metadata

Near real-time (as fast as they get into our database) data access methods allow the extraction of sensor data as time-series, either as processed scalar data with Quality Assurance and Control flags (QAQC) or directly as raw data obtained from the device in its specific output format. In contrast to the Data product download methods, this data can be downloaded directly without waiting for any kind of generation process.

Common use cases include:

  • Plotting time-series from properties in a specific timeframe or in "near real-time"
  • Quickly obtaining the latest reading from a particular sensor
  • Obtaining raw unprocessed data from our instruments (data might require processing to be readable)


The methods getDirectByLocation() and getDirectRawByLocation() obtain data readings from a location without knowing which device it came from (hence the need to specify a "device category code" instead of a single "device code"). You might want to obtain data by location instead of by device, as individual devices are often replaced/repositioned.

Each request to our API can return a maximum of 100,000 samples; larger data requests must be downloaded as a sequence of "pages". Use the "allPages" parameter to automatically download all pages required for your requested timeframe.

Summary

Method

Description

getDirectByLocation (filters, allPages)

Obtain scalar data readings from a device category in a locationlink
getDirectByDevice (filters, allPages)Obtain scalar data readings from a devicelink

getDirectRawByLocation (filters, allPages)

Obtain raw data readings from a device category in a locationlink

getDirectRawByDevice (filters, allPages)

Obtain raw data readings from a devicelink



getDirectByLocation

(Previously called getDirectScalar)

Downloads time-series of property readings from a device category in a location. The returned data contains a value, time and QAQC flag for every measurement in the time-series.

Parameter

Type

Description

Example

filtersdictionary

A dictionary of filters used to configure the data product request, to be provided to the getByLocation method from the API scalardata service.

The following filters are required and will set the data origin:

The following filters are optional:

Visit the getByLocation method documentation for information on filter usage.

{
'locationCode':'TWDP',
'deviceCategoryCode':'TSG',
'dateFrom':'2016-09-01T00:00:00.000Z',
'dateTo':'2016-09-01T08:00:00.000Z'
}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

  • False: Only download the first page of data
  • True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

Example: Get all property readings in a time range
from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'locationCode': 'TWDP',         # Tswwassen DP Ferry
    'deviceCategoryCode':'TSG',     # Thermosalinograph
    'dateFrom':'2016-09-01T00:00:00.000Z',
    'dateTo':'2016-09-01T00:01:00.000Z'
}
result = onc.getDirectByLocation(filters, allPages=True)
onc.print(result)

Returns

A dictionary with separate lists of sensor readings for each property, as returned by the scalardata getByLocation method.

By default, each property in the response includes a "data" dictionary with matching lists for "values", "sampleTimes" and "qaqcFlags".

Example response for a single property
{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?locationCode=TWDP&(...)",    
    "next": null,
    "sensorData": [
        {
            "sensorCategoryCode": "conductivity",
            "sensorCode": "Conductivity",
            "sensorName": "Conductivity",
            "unitOfMeasure": "S/m",
            "data": {
                "values": [3.442, 3.448, 3.448, 3.447],
                "sampleTimes": [
                    "2016-09-01T00:00:07.037Z",
                    "2016-09-01T00:00:17.041Z",
                    "2016-09-01T00:00:27.046Z",
                    "2016-09-01T00:00:37.040Z"
                ],
                "qaqcFlags": [1,1,1,1]
            }
        }
    ]
}

If the request filters include the "outputFormat" parameter set to "object", the data in "sensorData" for each sensor will be represented as a single list of self-contained objects as in the following example (Warning: This might noticeably increase the download size in long time-series):

Example response with "object" output format
{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?locationCode=TWDP&(...)",    
    "next": null,
    "sensorData": [
        {
            "data": [
                {
                    "qaqcFlag": 1,
                    "sampleTime": "2016-09-01T00:00:07.037Z",
                    "value": 28.9096
                },
                {
                    "qaqcFlag": 1,
                    "sampleTime": "2016-09-01T00:00:17.041Z",
                    "value": 28.9114
                },
                {
                    "qaqcFlag": 1,
                    "sampleTime": "2016-09-01T00:00:27.046Z",
                    "value": 28.8941
                },
                # (...)
            ]
        }
    ]
}

If the request filters include the "metadata" parameter set to "full", the response will also include a "metadata" dictionary as in the following example:

Example response with metadata
{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?locationCode=TWDP&(...)",
    "next": null,
    "sensorData": # (...),
    "metadata": {
        "boundingBox": {
            "maxDepth": 3.0,
            "maxLat": 49.004607,
            "maxLon": -123.185037,
            "minDepth": 3.0,
            "minLat": 49.004607,
            "minLon": -123.185037
        },
        "depth": 3.0,
        "deviceCategoryCode": "TSG",
        "lat": 49.004607,
        "locationName": "Tsawwassen - Duke Point",
        "lon": -123.185037
    }
}

If there are additional data pages to download, the "next" property will contain a dictionary with information to obtain the next page of data as in the following example:

Example response with a "next" page
{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?locationCode=TWDP&(...)",
    "next": {
            "locationCode": "TWDP",
            "deviceCategoryCode": "TSG",
            "dateFrom": "2016-09-01T00:00:37.040Z",
            "dateTo": "2016-09-01T00:01:00.000Z",
            "token": "YOUR_TOKEN",
            "method": "getByLocation"
    },
    "sensorData": # (...)
}
 Detailed description of returned data [+]

The response is a dictionary with the following keys:

KeyTypeDescription
queryUrlstringThe URL that obtained the results
sensorDatalistA list of dictionaries, one for each property obtained, with the data readings and sensor information
metaDatadictionary

A dictionary with general information from the instruments that contribute to the data readings returned.

Only included when the "metadata" request parameter has been set to "full"

nextdictionary

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data.

If this is the last page of data, "next" will be null.


Each "property" in the sensorData list is a dictionary with the following keys:

KeyTypeDescription
sensorCategoryCodefloatThe property code for this property
sensorCodestringThe code of the sensor that produced the property
sensorNamestringThe name of the sensor that produced the property
unitOfMeasurestringThe unit of measure for the property
data

dictionary

(or list)

By default, a dictionary with the readings for the property. The data for each sample is distributed into 3 lists of matching indexes: "values", "sampleTimes" and "qaqcFlags".

If the request filters include the "outputFormat" parameter set to "object", the "data" key will instead hold a list of self-contained dictionaries (one per sample), each with its own "value", "sampleTime" and "qaqcFlag" as explained in the scalardata getByLocation method documentation.

data.values

listList of values (float)

data.sampleTimes

listList with the sampling time for each corresponding element in the "values" list

data.qaqcFlags

listList with the QAQC flag for each corresponding element in the "values" list


When the "next" dictionary is not null (more data pages available), it will include the following keys:

KeyTypeDescription
urlstringThe URL to obtain the next data page
parametersdictionary

A complete dictionary of the filters used to build the "url" to the next data page.

Will usually contain exactly the same parameters provided in the original request, with the following exceptions:

  • "dateFrom" will be modified to match the start of the next page of data
  • If "dateTo" was provided as a relative date, it will be translated to an absolute date
  • In addition, includes the user "token" and the API "method" consumed ("getByLocation")
nextdictionary

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data.

If this is the last page of data, "next" will be null.


getDirectByDevice

Downloads time-series of property readings from a particular device. The returned data contains a value, time and QAQC flag for every measurement in the time-series.

Parameter

Type

Description

Example

filtersdictionary

A dictionary of filters used to configure the data product request, to be provided to the getByDevice method from the API scalardata service.

The deviceCode filter is required and will set the data origin.

The following filters are optional:

Visit the getByDevice method documentation for information on filter usage.

{
'deviceCode': 'AMLMETRECX50348',
'dateFrom':'2019-06-01T00:00:00.000Z',
'dateTo':'2019-06-01T00:00:10.000Z'

}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

  • False: Only download the first page of data
  • True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

Example: Get all readings from a specific device in a time range
from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'deviceCode': 'AMLMETRECX50348',
    'dateFrom':'2019-06-01T00:00:00.000Z',
    'dateTo':'2019-06-01T00:00:10.000Z'
}
result = onc.getDirectByDevice(filters)
onc.print(result)

Returns

A dictionary with separate lists of sensor readings for each property, as returned by the scalardata getByDevice method.

By default, each property in the response includes a "data" dictionary with matching lists for "values", "sampleTimes" and "qaqcFlags".

Example response for a single property
{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?method=getByDevice&(...)",    
    "next": null,
    "sensorData": [
        {
            "sensorCategoryCode": "conductivity",
            "sensorCode": "Conductivity",
            "sensorName": "Conductivity",
            "unitOfMeasure": "S/m",
            "data": {
                "values": [3.442, 3.448, 3.448, 3.447],
                "sampleTimes": [
                    "2016-09-01T00:00:07.037Z",
                    "2016-09-01T00:00:17.041Z",
                    "2016-09-01T00:00:27.046Z",
                    "2016-09-01T00:00:37.040Z"
                ],
                "qaqcFlags": [1,1,1,1]
            }
        }
    ]
}

If the request filters include the "outputFormat" parameter set to "object", the data in "sensorData" for each sensor will be represented as a single list of self-contained objects in the same way as explained for the getDirectByLocation method response (Warning: This might noticeably increase the download size in long time-series).

If the request filters include the "metadata" parameter set to "full", the response will also include a "metadata" dictionary in the same way as explained for the getDirectByLocation method response.

If there are additional data pages to download, the "next" property will contain a dictionary with information to obtain the next page of data in the same way as explained for the getDirectByLocation method response.

 Detailed description of returned data [+]

The response is a dictionary with the following keys:

KeyTypeDescription
queryUrlstringThe URL that obtained the results
sensorDatalistA list of dictionaries, one for each property obtained, with the data readings and sensor information
metaDatadictionary

A dictionary with general information from the instruments that contribute to the data readings returned.

Only included when the "metadata" request parameter has been set to "full"

nextdictionary

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data.

If this is the last page of data, "next" will be null.


Each "property" in the sensorData list is a dictionary with the following keys:

KeyTypeDescription
sensorCategoryCodefloatThe property code for this property
sensorCodestringThe code of the sensor that produced the property
sensorNamestringThe name of the sensor that produced the property
unitOfMeasurestringThe unit of measure for the property
data

dictionary

(or list)

By default, a dictionary with the readings for the property. The data for each sample is distributed into 3 lists of matching indexes: "values", "sampleTimes" and "qaqcFlags".

If the request filters include the "outputFormat" parameter set to "object", the "data" key will instead hold a list of self-contained dictionaries (one per sample), each with its own "value", "sampleTime" and "qaqcFlag" as explained in the scalardata getByLocation method documentation.

data.values

listList of values (float)

data.sampleTimes

listList with the sampling time for each corresponding element in the "values" list

data.qaqcFlags

listList with the QAQC flag for each corresponding element in the "values" list


When the "next" dictionary is not null (more data pages available), it will include the following keys:

KeyTypeDescription
urlstringThe URL to obtain the next data page
parametersdictionary

A complete dictionary of the filters used to build the "url" to the next data page.

Will usually contain exactly the same parameters provided in the original request, with the following exceptions:

  • "dateFrom" will be modified to match the start of the next page of data
  • If "dateTo" was provided as a relative date, it will be translated to an absolute date
  • In addition, includes the user "token" and the API "method" consumed ("getByLocation")
nextdictionary

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data.

If this is the last page of data, "next" will be null.


getDirectRawByLocation

Downloads time-series of raw sensor readings from a device category in a location. The response will contain readings for all properties available in the devices providing the data.

Depending on the output format of the source devices, raw data might not be in a readable format without additional processing.

Parameter

Type

Description

Example

filtersdictionary

A dictionary of filters used to configure the data product request, to be provided to the rawdata getByLocation API method.

The following filters are required and will set the data origin:

The following filters are optional:

Visit the getByLocation method documentation for information on filter usage.

{
'locationCode':'TWDP',
'deviceCategoryCode':'TSG',
'dateFrom':'2016-09-01T00:00:00.000Z',
'dateTo':'2016-09-01T08:00:00.000Z'
}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

  • False: Only download the first page of data
  • True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

Example: Get raw readings from a location in a time range
from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'locationCode': 'TWDP',         # Tswwassen DP Ferry
    'deviceCategoryCode':'TSG',     # Thermosalinograph
    'dateFrom':'2016-09-01T00:00:00.000Z',
    'dateTo':'2016-09-01T00:01:00.000Z'
}
result = onc.getDirectRawByLocation(filters, allPages=True)
onc.print(result)

Returns

A dictionary with separate lists of raw device output. Each data sample in the result is represented in matching lists for "readings", "times" and "lineTypes",  as returned by the rawdata getByLocation method.

By default, raw readings will be returned in a structure as presented in the following example:

Example raw response
{
    "data": {
        "readings": [
            "20160901-00:00:07.037, 12.9162,  3.44294,  28.9096",
            "20160901-00:00:17.041, 12.9798,  3.44834,  28.9114",
            "20160901-00:00:27.046, 13.0016,  3.44827,  28.8941",
            "20160901-00:00:37.040, 13.0094,  3.44771,  28.8829",
            "20160901-00:00:47.045, 13.0038,  3.44889,  28.8981"
        ],
        "times": [
            "2016-09-01T00:00:07.037Z",
            "2016-09-01T00:00:17.041Z",
            "2016-09-01T00:00:27.046Z",
            "2016-09-01T00:00:37.040Z",
            "2016-09-01T00:00:47.045Z"
        ],
        "lineTypes": [" "," "," "," "]
    },
    "metadata": {
        "locationName": "Tsawwassen - Duke Point"
    },
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/rawdata?locationCode=TWDP&(...)"
}

If the request filters include the "outputFormat" parameter set to "object", the data returned will be represented as a single list of self-contained objects as in the following example (Warning: This might noticeably increase the download size in long time-series):

Example raw response with "object" format
{
   "data": [
        {
            "lineType": " ",
            "rawData": "20160901-00:00:07.037, 12.9162,  3.44294,  28.9096",
            "sampleTime": "2016-09-01T00:00:07.037Z"
        },
        {
            "lineType": " ",
            "rawData": "20160901-00:00:17.041, 12.9798,  3.44834,  28.9114",
            "sampleTime": "2016-09-01T00:00:17.041Z"
        },
        {
            "lineType": " ",
            "rawData": "20160901-00:00:27.046, 13.0016,  3.44827,  28.8941",
            "sampleTime": "2016-09-01T00:00:27.046Z"
        },
        (...)
    ],
    "metadata": {
        "locationName": "Tsawwassen - Duke Point"
    },
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/rawdata?locationCode=TWDP&(...)"
}

If there are additional data pages to download, the "next" property will contain a dictionary with information to obtain the next page of data, just as explained for the getDirectScalar method.

 Detailed description of returned data [+]

The response is a dictionary with the following keys:

KeyTypeDescription
queryUrlstringThe URL that obtained the results
data

dictionary

(or list)

By default, a dictionary with the raw readings obtained. The data for each reading is distributed into 3 lists of matching indexes: "readings", "times" and "lineTypes".

If the request filters include the "outputFormat" parameter set to "object", the "data" key will instead hold a list of self-contained dictionaries (one per sample), each with its own "reading", "time" and "lineType" as explained in the rawdata getByLocation documentation.

data.readings

listList of raw readings. The readings might be strings or integers depending on the raw data type.

data.times

listList with the sampling time for each corresponding element in the "readings" list

data.lineTypes

list

List of (string) with the type of raw reading line for each corresponding element in the "readings" list.

The line type " " matches data readings and ">" and "<" are used for command lines.

metadatadictionaryA dictionary with information on the data origin

metadata.locationName

stringThe name of the location the data originates from
nextdictionary

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data, as explained for the getDirectScalar method.

If this is the last page of data, "next" will be null.


getDirectRawByDevice

Downloads time-series of raw sensor readings from a unique device. The response will contain readings for all properties available in the device.

Depending on the output format of the device, raw data might not be in a readable format without additional processing (for more information, consult the device documentation in the Data Search tool).

Parameter

Type

Description

Example

filtersdictionary

A dictionary of filters used to configure the data product request, to be provided to the rawdata getByDevice API method.

The filter deviceCode is required and will set the data origin.

The following filters are optional:

  • dateFrom
  • dateTo
  • rowLimit
  • sizeLimit
  • convertHexToDecimal
  • outputFormat
  • getLatest

Visit the getByDevice method documentation for information on filter usage.

{
'deviceCode':'AMLMETRECX50348',
'dateFrom':'2019-06-01T00:00:00.000Z',
'dateTo':'2019-06-01T00:00:05.000Z'
}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

  • False: Only download the first page of data
  • True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

Example: Get raw readings from a device in a time range
from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'deviceCode':'AMLMETRECX50348',      # A CTD in Burrard Inlet
    'dateFrom':'2019-06-01T00:00:00.000Z',
    'dateTo':'2019-06-01T00:00:05.000Z'
}
result = onc.getDirectRawByDevice(filters, allPages=False)
onc.print(result)

Returns

A dictionary with separate lists of raw device output. Each data sample in the result is represented in matching lists for "readings", "times" and "lineTypes",  as returned by the rawdata getByDevice method.

By default, raw readings will be returned in a structure as presented in the following example:

Example raw response
{
    "data": {
        "readings": [
            "20160901-00:00:07.037, 12.9162,  3.44294,  28.9096",
            "20160901-00:00:17.041, 12.9798,  3.44834,  28.9114",
            "20160901-00:00:27.046, 13.0016,  3.44827,  28.8941",
            "20160901-00:00:37.040, 13.0094,  3.44771,  28.8829",
            "20160901-00:00:47.045, 13.0038,  3.44889,  28.8981"
        ],
        "times": [
            "2016-09-01T00:00:07.037Z",
            "2016-09-01T00:00:17.041Z",
            "2016-09-01T00:00:27.046Z",
            "2016-09-01T00:00:37.040Z",
            "2016-09-01T00:00:47.045Z"
        ],
        "lineTypes": [" "," "," "," "]
    },
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/rawdata?deviceCode=AMLMETRECX5034&(...)"
}

If the request filters include the "outputFormat" parameter set to "object", the data returned will be represented as a single list of self-contained objects as in the following example (Warning: This might noticeably increase the download size in long time-series):

Example raw response with "object" format
{
   "data": [
        {
            "lineType": " ",
            "rawData": "20160901-00:00:07.037, 12.9162,  3.44294,  28.9096",
            "sampleTime": "2016-09-01T00:00:07.037Z"
        },
        {
            "lineType": " ",
            "rawData": "20160901-00:00:17.041, 12.9798,  3.44834,  28.9114",
            "sampleTime": "2016-09-01T00:00:17.041Z"
        },
        {
            "lineType": " ",
            "rawData": "20160901-00:00:27.046, 13.0016,  3.44827,  28.8941",
            "sampleTime": "2016-09-01T00:00:27.046Z"
        },
        (...)
    ],
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/rawdata?deviceCode=AMLMETRECX5034&(...)"
}

If there are additional data pages to download, the "next" property will contain a dictionary with information to obtain the next page of data, just as explained for the getDirectScalar method.

 Detailed description of returned data [+]

The response is a dictionary with the following keys:

KeyTypeDescription
queryUrlstringThe URL that obtained the results
data

dictionary

(or list)

By default, a dictionary with the raw readings obtained. The data for each reading is distributed into 3 lists of matching indexes: "readings", "times" and "lineTypes".

If the request filters include the "outputFormat" parameter set to "object", the "data" key will instead hold a list of self-contained dictionaries (one per sample), each with its own "reading", "time" and "lineType" as explained in the rawdata getByLocation documentation.

data.readings

listList of raw readings. The readings might be strings or integers depending on the raw data type.

data.times

listList with the sampling time for each corresponding element in the "readings" list

data.lineTypes

list

List of (string) with the type of raw reading line for each corresponding element in the "readings" list.

The line type " " matches data readings and ">" and "<" are used for command lines.

nextdictionary

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data, just as explained for the getDirectScalar method.

If this is the last page of data, "next" will be null.

  • No labels