Near real-time data access methods

Near real-time (as fast as they get into our database) data access methods allow the extraction of sensor data as time-series, either as processed scalar data with Quality Assurance and Control flags (QAQC) or directly as raw data obtained from the device in its specific output format. In contrast to the Data product download methods, this data can be downloaded directly without waiting for any kind of generation process.

Common use cases include:

Plotting time-series from properties in a specific timeframe or in "near real-time"
Quickly obtaining the latest reading from a particular sensor
Obtaining raw unprocessed data from our instruments (data might require processing to be readable)

The methods getDirectByLocation() and getDirectRawByLocation() obtain data readings from a location without knowing which device it came from (hence the need to specify a "device category code" instead of a single "device code"). You might want to obtain data by location instead of by device, as individual devices are often replaced/repositioned.

Each request to our API can return a maximum of 100,000 samples; larger data requests must be downloaded as a sequence of "pages". Use the "allPages" parameter to automatically download all pages required for your requested timeframe.

Summary

Method	Description
`getDirectByLocation (filters, allPages)`	Obtain scalar data readings from a device category in a location	link
`getDirectByDevice (filters, allPages)`	Obtain scalar data readings from a device	link
`getDirectRawByLocation (filters, allPages)`	Obtain raw data readings from a device category in a location	link
`getDirectRawByDevice (filters, allPages)`	Obtain raw data readings from a device	link

getDirectByLocation

(Previously called getDirectScalar)

Downloads time-series of property readings from a device category in a location. The returned data contains a value, time and QAQC flag for every measurement in the time-series.

Parameter

Type

Description

Example

filters

dictionary

A dictionary of filters used to configure the data product request, to be provided to the getByLocation method from the API scalardata service.

The following filters are required and will set the data origin:

The following filters are optional:

propertyCode
dateFrom
dateTo
metadata
rowLimit
outputFormat
getLatest

Visit the getByLocation method documentation for information on filter usage.

{
'locationCode':'TWDP',
'deviceCategoryCode':'TSG',
'dateFrom':'2016-09-01T00:00:00.000Z',
'dateTo':'2016-09-01T08:00:00.000Z'
}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

False: Only download the first page of data
True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

Example: Get all property readings in a time range

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'locationCode': 'TWDP',         # Tswwassen DP Ferry
    'deviceCategoryCode':'TSG',     # Thermosalinograph
    'dateFrom':'2016-09-01T00:00:00.000Z',
    'dateTo':'2016-09-01T00:01:00.000Z'
}
result = onc.getDirectByLocation(filters, allPages=True)
onc.print(result)

Returns

A dictionary with separate lists of sensor readings for each property, as returned by the scalardata getByLocation method.

By default, each property in the response includes a "data" dictionary with matching lists for "values", "sampleTimes" and "qaqcFlags".

Example response for a single property

{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?locationCode=TWDP&(...)",    
    "next": null,
    "sensorData": [
        {
            "sensorCategoryCode": "conductivity",
            "sensorCode": "Conductivity",
            "sensorName": "Conductivity",
            "unitOfMeasure": "S/m",
            "data": {
                "values": [3.442, 3.448, 3.448, 3.447],
                "sampleTimes": [
                    "2016-09-01T00:00:07.037Z",
                    "2016-09-01T00:00:17.041Z",
                    "2016-09-01T00:00:27.046Z",
                    "2016-09-01T00:00:37.040Z"
                ],
                "qaqcFlags": [1,1,1,1]
            }
        }
    ]
}

If the request filters include the "outputFormat" parameter set to "object", the data in "sensorData" for each sensor will be represented as a single list of self-contained objects as in the following example (Warning: This might noticeably increase the download size in long time-series):

Example response with "object" output format

{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?locationCode=TWDP&(...)",    
    "next": null,
    "sensorData": [
        {
            "data": [
                {
                    "qaqcFlag": 1,
                    "sampleTime": "2016-09-01T00:00:07.037Z",
                    "value": 28.9096
                },
                {
                    "qaqcFlag": 1,
                    "sampleTime": "2016-09-01T00:00:17.041Z",
                    "value": 28.9114
                },
                {
                    "qaqcFlag": 1,
                    "sampleTime": "2016-09-01T00:00:27.046Z",
                    "value": 28.8941
                },
                # (...)
            ]
        }
    ]
}

If the request filters include the "metadata" parameter set to "full", the response will also include a "metadata" dictionary as in the following example:

Example response with metadata

{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?locationCode=TWDP&(...)",
    "next": null,
    "sensorData": # (...),
    "metadata": {
        "boundingBox": {
            "maxDepth": 3.0,
            "maxLat": 49.004607,
            "maxLon": -123.185037,
            "minDepth": 3.0,
            "minLat": 49.004607,
            "minLon": -123.185037
        },
        "depth": 3.0,
        "deviceCategoryCode": "TSG",
        "lat": 49.004607,
        "locationName": "Tsawwassen - Duke Point",
        "lon": -123.185037
    }
}

If there are additional data pages to download, the "next" property will contain a dictionary with information to obtain the next page of data as in the following example:

Example response with a "next" page

{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?locationCode=TWDP&(...)",
    "next": {
            "locationCode": "TWDP",
            "deviceCategoryCode": "TSG",
            "dateFrom": "2016-09-01T00:00:37.040Z",
            "dateTo": "2016-09-01T00:01:00.000Z",
            "token": "YOUR_TOKEN",
            "method": "getByLocation"
    },
    "sensorData": # (...)
}

Detailed description of returned data [+]

The response is a dictionary with the following keys:

Key	Type	Description
queryUrl	string	The URL that obtained the results
sensorData	list	A list of dictionaries, one for each property obtained, with the data readings and sensor information
metaData	dictionary	A dictionary with general information from the instruments that contribute to the data readings returned. Only included when the "`metadata`" request parameter has been set to "`full`"
next	dictionary	If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data. If this is the last page of data, "`next`" will be null.

Each "property" in the sensorData list is a dictionary with the following keys:

Key	Type	Description
sensorCategoryCode	float	The property code for this property
sensorCode	string	The code of the sensor that produced the property
sensorName	string	The name of the sensor that produced the property
unitOfMeasure	string	The unit of measure for the property
data	dictionary (or list)	By default, a dictionary with the readings for the property. The data for each sample is distributed into 3 lists of matching indexes: "`values`", "`sampleTimes`" and "`qaqcFlags`". If the request filters include the "`outputFormat`" parameter set to "`object`", the "`data`" key will instead hold a list of self-contained dictionaries (one per sample), each with its own "`value`", "`sampleTime`" and "`qaqcFlag`" as explained in the scalardata getByLocation method documentation.
data.values	list	List of values (float)
data.sampleTimes	list	List with the sampling time for each corresponding element in the "`values`" list
data.qaqcFlags	list	List with the QAQC flag for each corresponding element in the "`values`" list

When the "next" dictionary is not null (more data pages available), it will include the following keys:

Key Type Description

url string The URL to obtain the next data page

parameters

dictionary

A complete dictionary of the filters used to build the "url" to the next data page.

Will usually contain exactly the same parameters provided in the original request, with the following exceptions:

"dateFrom" will be modified to match the start of the next page of data
If "dateTo" was provided as a relative date, it will be translated to an absolute date
In addition, includes the user "token" and the API "method" consumed ("getByLocation")

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data.

If this is the last page of data, "next" will be null.

getDirectByDevice

Downloads time-series of property readings from a particular device. The returned data contains a value, time and QAQC flag for every measurement in the time-series.

Parameter

Type

Description

Example

filters

dictionary

A dictionary of filters used to configure the data product request, to be provided to the getByDevice method from the API scalardata service.

The deviceCode filter is required and will set the data origin.

The following filters are optional:

dateFrom
dateTo
rowLimit
outputFormat
getLatest

Visit the getByDevice method documentation for information on filter usage.

{'deviceCode': 'AMLMETRECX50348', 'dateFrom':'2019-06-01T00:00:00.000Z', 'dateTo':'2019-06-01T00:00:10.000Z'
}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

False: Only download the first page of data
True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

Example: Get all readings from a specific device in a time range

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'deviceCode': 'AMLMETRECX50348',
    'dateFrom':'2019-06-01T00:00:00.000Z',
    'dateTo':'2019-06-01T00:00:10.000Z'
}
result = onc.getDirectByDevice(filters)
onc.print(result)

Returns

A dictionary with separate lists of sensor readings for each property, as returned by the scalardata getByDevice method.

By default, each property in the response includes a "data" dictionary with matching lists for "values", "sampleTimes" and "qaqcFlags".

Example response for a single property

{
    "queryUrl": "https://data.oceannetworks.ca/api/scalardata?method=getByDevice&(...)",    
    "next": null,
    "sensorData": [
        {
            "sensorCategoryCode": "conductivity",
            "sensorCode": "Conductivity",
            "sensorName": "Conductivity",
            "unitOfMeasure": "S/m",
            "data": {
                "values": [3.442, 3.448, 3.448, 3.447],
                "sampleTimes": [
                    "2016-09-01T00:00:07.037Z",
                    "2016-09-01T00:00:17.041Z",
                    "2016-09-01T00:00:27.046Z",
                    "2016-09-01T00:00:37.040Z"
                ],
                "qaqcFlags": [1,1,1,1]
            }
        }
    ]
}

If the request filters include the "outputFormat" parameter set to "object", the data in "sensorData" for each sensor will be represented as a single list of self-contained objects in the same way as explained for the getDirectByLocation method response (Warning: This might noticeably increase the download size in long time-series).

If the request filters include the "metadata" parameter set to "full", the response will also include a "metadata" dictionary in the same way as explained for the getDirectByLocation method response.

If there are additional data pages to download, the "next" property will contain a dictionary with information to obtain the next page of data in the same way as explained for the getDirectByLocation method response.

Detailed description of returned data [+]

The response is a dictionary with the following keys:

Key	Type	Description
queryUrl	string	The URL that obtained the results
sensorData	list	A list of dictionaries, one for each property obtained, with the data readings and sensor information
metaData	dictionary	A dictionary with general information from the instruments that contribute to the data readings returned. Only included when the "`metadata`" request parameter has been set to "`full`"
next	dictionary	If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data. If this is the last page of data, "`next`" will be null.

Each "property" in the sensorData list is a dictionary with the following keys:

Key	Type	Description
sensorCategoryCode	float	The property code for this property
sensorCode	string	The code of the sensor that produced the property
sensorName	string	The name of the sensor that produced the property
unitOfMeasure	string	The unit of measure for the property
data	dictionary (or list)	By default, a dictionary with the readings for the property. The data for each sample is distributed into 3 lists of matching indexes: "`values`", "`sampleTimes`" and "`qaqcFlags`". If the request filters include the "`outputFormat`" parameter set to "`object`", the "`data`" key will instead hold a list of self-contained dictionaries (one per sample), each with its own "`value`", "`sampleTime`" and "`qaqcFlag`" as explained in the scalardata getByLocation method documentation.
data.values	list	List of values (float)
data.sampleTimes	list	List with the sampling time for each corresponding element in the "`values`" list
data.qaqcFlags	list	List with the QAQC flag for each corresponding element in the "`values`" list

When the "next" dictionary is not null (more data pages available), it will include the following keys:

Key Type Description

url string The URL to obtain the next data page

parameters

dictionary

A complete dictionary of the filters used to build the "url" to the next data page.

Will usually contain exactly the same parameters provided in the original request, with the following exceptions:

"dateFrom" will be modified to match the start of the next page of data
If "dateTo" was provided as a relative date, it will be translated to an absolute date
In addition, includes the user "token" and the API "method" consumed ("getByLocation")

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data.

If this is the last page of data, "next" will be null.

getDirectRawByLocation

Downloads time-series of raw sensor readings from a device category in a location. The response will contain readings for all properties available in the devices providing the data.

Depending on the output format of the source devices, raw data might not be in a readable format without additional processing.

Parameter

Type

Description

Example

filters

dictionary

A dictionary of filters used to configure the data product request, to be provided to the rawdata getByLocation API method.

The following filters are required and will set the data origin:

The following filters are optional:

dateFrom
dateTo
metadata
rowLimit
sizeLimit
outputFormat
getLatest

Visit the getByLocation method documentation for information on filter usage.

{
'locationCode':'TWDP',
'deviceCategoryCode':'TSG',
'dateFrom':'2016-09-01T00:00:00.000Z',
'dateTo':'2016-09-01T08:00:00.000Z'
}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

False: Only download the first page of data
True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

Example: Get raw readings from a location in a time range

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'locationCode': 'TWDP',         # Tswwassen DP Ferry
    'deviceCategoryCode':'TSG',     # Thermosalinograph
    'dateFrom':'2016-09-01T00:00:00.000Z',
    'dateTo':'2016-09-01T00:01:00.000Z'
}
result = onc.getDirectRawByLocation(filters, allPages=True)
onc.print(result)

Returns

A dictionary with separate lists of raw device output. Each data sample in the result is represented in matching lists for "readings", "times" and "lineTypes", as returned by the rawdata getByLocation method.

By default, raw readings will be returned in a structure as presented in the following example:

Example raw response

{
    "data": {
        "readings": [
            "20160901-00:00:07.037, 12.9162,  3.44294,  28.9096",
            "20160901-00:00:17.041, 12.9798,  3.44834,  28.9114",
            "20160901-00:00:27.046, 13.0016,  3.44827,  28.8941",
            "20160901-00:00:37.040, 13.0094,  3.44771,  28.8829",
            "20160901-00:00:47.045, 13.0038,  3.44889,  28.8981"
        ],
        "times": [
            "2016-09-01T00:00:07.037Z",
            "2016-09-01T00:00:17.041Z",
            "2016-09-01T00:00:27.046Z",
            "2016-09-01T00:00:37.040Z",
            "2016-09-01T00:00:47.045Z"
        ],
        "lineTypes": [" "," "," "," "]
    },
    "metadata": {
        "locationName": "Tsawwassen - Duke Point"
    },
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/rawdata?locationCode=TWDP&(...)"
}

If the request filters include the "outputFormat" parameter set to "object", the data returned will be represented as a single list of self-contained objects as in the following example (Warning: This might noticeably increase the download size in long time-series):

Example raw response with "object" format

{
   "data": [
        {
            "lineType": " ",
            "rawData": "20160901-00:00:07.037, 12.9162,  3.44294,  28.9096",
            "sampleTime": "2016-09-01T00:00:07.037Z"
        },
        {
            "lineType": " ",
            "rawData": "20160901-00:00:17.041, 12.9798,  3.44834,  28.9114",
            "sampleTime": "2016-09-01T00:00:17.041Z"
        },
        {
            "lineType": " ",
            "rawData": "20160901-00:00:27.046, 13.0016,  3.44827,  28.8941",
            "sampleTime": "2016-09-01T00:00:27.046Z"
        },
        (...)
    ],
    "metadata": {
        "locationName": "Tsawwassen - Duke Point"
    },
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/rawdata?locationCode=TWDP&(...)"
}

If there are additional data pages to download, the "next" property will contain a dictionary with information to obtain the next page of data, just as explained for the getDirectScalar method.

Detailed description of returned data [+]

The response is a dictionary with the following keys:

Key	Type	Description
queryUrl	string	The URL that obtained the results
data	dictionary (or list)	By default, a dictionary with the raw readings obtained. The data for each reading is distributed into 3 lists of matching indexes: "`readings`", "`times`" and "`lineTypes`". If the request filters include the "`outputFormat`" parameter set to "`object`", the "`data`" key will instead hold a list of self-contained dictionaries (one per sample), each with its own "`reading`", "`time`" and "`lineType`" as explained in the rawdata getByLocation documentation.
data.readings	list	List of raw readings. The readings might be strings or integers depending on the raw data type.
data.times	list	List with the sampling time for each corresponding element in the "`readings`" list
data.lineTypes	list	List of (string) with the type of raw reading line for each corresponding element in the "`readings`" list. The line type " " matches data readings and ">" and "<" are used for command lines.
metadata	dictionary	A dictionary with information on the data origin
metadata.locationName	string	The name of the location the data originates from
next	dictionary	If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data, as explained for the getDirectScalar method. If this is the last page of data, "`next`" will be null.

getDirectRawByDevice

Downloads time-series of raw sensor readings from a unique device. The response will contain readings for all properties available in the device.

Depending on the output format of the device, raw data might not be in a readable format without additional processing (for more information, consult the device documentation in the Data Search tool).

Parameter

Type

Description

Example

filters

dictionary

A dictionary of filters used to configure the data product request, to be provided to the rawdata getByDevice API method.

The filter deviceCode is required and will set the data origin.

The following filters are optional:

dateFrom
dateTo
rowLimit
sizeLimit
convertHexToDecimal
outputFormat
getLatest

Visit the getByDevice method documentation for information on filter usage.

{
'deviceCode':'AMLMETRECX50348',
'dateFrom':'2019-06-01T00:00:00.000Z',
'dateTo':'2019-06-01T00:00:05.000Z'
}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

False: Only download the first page of data
True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

Example: Get raw readings from a device in a time range

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'deviceCode':'AMLMETRECX50348',      # A CTD in Burrard Inlet
    'dateFrom':'2019-06-01T00:00:00.000Z',
    'dateTo':'2019-06-01T00:00:05.000Z'
}
result = onc.getDirectRawByDevice(filters, allPages=False)
onc.print(result)

Returns

A dictionary with separate lists of raw device output. Each data sample in the result is represented in matching lists for "readings", "times" and "lineTypes", as returned by the rawdata getByDevice method.

By default, raw readings will be returned in a structure as presented in the following example:

Example raw response

{
    "data": {
        "readings": [
            "20160901-00:00:07.037, 12.9162,  3.44294,  28.9096",
            "20160901-00:00:17.041, 12.9798,  3.44834,  28.9114",
            "20160901-00:00:27.046, 13.0016,  3.44827,  28.8941",
            "20160901-00:00:37.040, 13.0094,  3.44771,  28.8829",
            "20160901-00:00:47.045, 13.0038,  3.44889,  28.8981"
        ],
        "times": [
            "2016-09-01T00:00:07.037Z",
            "2016-09-01T00:00:17.041Z",
            "2016-09-01T00:00:27.046Z",
            "2016-09-01T00:00:37.040Z",
            "2016-09-01T00:00:47.045Z"
        ],
        "lineTypes": [" "," "," "," "]
    },
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/rawdata?deviceCode=AMLMETRECX5034&(...)"
}

If the request filters include the "outputFormat" parameter set to "object", the data returned will be represented as a single list of self-contained objects as in the following example (Warning: This might noticeably increase the download size in long time-series):

Example raw response with "object" format

{
   "data": [
        {
            "lineType": " ",
            "rawData": "20160901-00:00:07.037, 12.9162,  3.44294,  28.9096",
            "sampleTime": "2016-09-01T00:00:07.037Z"
        },
        {
            "lineType": " ",
            "rawData": "20160901-00:00:17.041, 12.9798,  3.44834,  28.9114",
            "sampleTime": "2016-09-01T00:00:17.041Z"
        },
        {
            "lineType": " ",
            "rawData": "20160901-00:00:27.046, 13.0016,  3.44827,  28.8941",
            "sampleTime": "2016-09-01T00:00:27.046Z"
        },
        (...)
    ],
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/rawdata?deviceCode=AMLMETRECX5034&(...)"
}

If there are additional data pages to download, the "next" property will contain a dictionary with information to obtain the next page of data, just as explained for the getDirectScalar method.

Detailed description of returned data [+]

The response is a dictionary with the following keys:

Key	Type	Description
queryUrl	string	The URL that obtained the results
data	dictionary (or list)	By default, a dictionary with the raw readings obtained. The data for each reading is distributed into 3 lists of matching indexes: "`readings`", "`times`" and "`lineTypes`". If the request filters include the "`outputFormat`" parameter set to "`object`", the "`data`" key will instead hold a list of self-contained dictionaries (one per sample), each with its own "`reading`", "`time`" and "`lineType`" as explained in the rawdata getByLocation documentation.
data.readings	list	List of raw readings. The readings might be strings or integers depending on the raw data type.
data.times	list	List with the sampling time for each corresponding element in the "`readings`" list
data.lineTypes	list	List of (string) with the type of raw reading line for each corresponding element in the "`readings`" list. The line type " " matches data readings and ">" and "<" are used for command lines.
next	dictionary	If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data, just as explained for the getDirectScalar method. If this is the last page of data, "`next`" will be null.

Page tree

Near real-time data access methods

Summary

getDirectByLocation

Returns

getDirectByDevice

Returns

getDirectRawByLocation

Returns

getDirectRawByDevice

Returns