Data product download methods

These methods allow you to request and download more than 120 different types of ONC data products, with granular control over what data to obtain, from where, and in what time frame. They are comparable to the download functionality from ONC's Data Search tool.

Examples of usage include:

Downloading PNG plots of sensor readings in a device
Downloading sensor readings as .mat files, text files or in commercial manufacturer formats
Downloading compressed or raw audio files from hydrophones

If the data product requested doesn't exist in our archive, it will be generated by our servers before your download starts.

Summary

Method	Description
`orderDataProduct (filters, maxRetries, downloadResultsOnly, includeMetadataFile)`	Request, run and download a data product	link
`requestDataProduct (filters)`	Request a data product	link
`runDataProduct (dpRequestId, waitComplete)`	Run a data product request	link
`downloadDataProduct (runId, maxRetries, downloadResultsOnly, includeMetadataFile, overwrite)`	Download an available data product	link

orderDataProduct

Request, run and download a data product.

Downloads files to the outPath directory as specified in the ONC class constructor ("output" directory by default). The download directory will be created automatically if it doesn't exist.

This method executes the data product download workflow described in the API Data Product Delivery Service.

Parameter

Type

Description

Example

filters

^filters

A group of filters used to configure the data product request, to be provided to the request method from the Data Product Delivery service.

The data source is specified with one of the following sets of filters:

Data source	Filters to include
A device	deviceCode
Property in a device	propertyCode, deviceCode
Property in a location	locationCode, propertyCode
Device category in a location	locationCode, deviceCategoryCode
Property in a device category in a location	locationCode, deviceCategoryCode, propertyCode

In addition, the following filters are mandatory:

Filters must also include the data product options for the requested data product; Otherwise, the API will return an error with instructions on what filters to add.

{
'locationCode': 'BACAX',

'deviceCategoryCode': 'ADCP2MHZ',
'dataProductCode': 'TSSD',
'extension': 'csv',
'dateFrom':'2016-07-27T00:00:00.000Z',
'dateTo': '2016-08-01T00:00:00.000Z',
'dpo_qualityControl': 1,
'dpo_resample': 'none',

'dpo_dataGaps': 0
}

{
'locationCode', 'BACAX',

'deviceCategoryCode', 'ADCP2MHZ',
'dataProductCode', 'TSSD',
'extension', 'csv',
'dateFrom','2016-07-27T00:00:00.000Z',
'dateTo', '2016-08-01T00:00:00.000Z',
'dpo_qualityControl', 1,
'dpo_resample', 'none',

'dpo_dataGaps', 0
}

list(
'locationCode' = 'BACAX',

'deviceCategoryCode' = 'ADCP2MHZ',
'dataProductCode' = 'TSSD',
'extension' = 'csv',
'dateFrom'='2016-07-27T00:00:00.000Z',
'dateTo' = '2016-08-01T00:00:00.000Z',
'dpo_qualityControl' = 1,
'dpo_resample' = 'none',

'dpo_dataGaps' = 0
)

maxRetries

int

The number of times to poll the service asking if the product is ready for download, before the method aborts.

Default: 0 (no limit)

1000

downloadResultsOnly

boolean

Whether the files will be downloaded or if only the download URL for each file will be returned.

^True: Files will not be downloaded. The file download URLs will be included in the method results
^False: Files are downloaded to the output path

Default: ^False

^False

includeMetadataFile

boolean

Indicates if the metadata file associated with the data product request will be downloaded.

^True: Metadata file will be downloaded
^False: Metadata file will not be downloaded

Default: ^False

^False

overwrite

boolean

Whether new files downloaded will overwrite previous files with the same filename found in the output directory.

^True: Overwrite files with the same filename
^False: Do not overwrite files (downloaded files with the same filename will be skipped)

Default: ^False

^False

(parameters with an underline are required)

Example: Download a PNG plot for all properties in a device category in a location

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
	'dataProductCode'   : 'TSSP',
	'extension'         : 'png',
	'locationCode'      : 'CRIP.C1',
	'deviceCategoryCode': 'CTD',
	'dateFrom'          : '2019-03-20T00:00:00.000Z',
	'dateTo'            : '2019-03-20T00:30:00.000Z',
	'dpo_qualityControl': '1',
	'dpo_resample'      : 'none'
}
result = onc.orderDataProduct(filters)

onc.print(result)

onc = Onc(YOUR_TOKEN_HERE);

filters = {
	'dataProductCode',    'TSSP',
	'extension',          'png',
	'locationCode',       'CRIP.C1',
	'deviceCategoryCode', 'CTD',
	'dateFrom',           '2019-03-20T00:00:00.000Z',
	'dateTo',             '2019-03-20T00:10:00.000Z',
	'dpo_qualityControl', '1',
	'dpo_resample',       'none'
};
result = onc.orderDataProduct(filters);

onc.print(result);

library(onc)
onc <- Onc(YOUR_TOKEN_HERE)

filters = list(
	'dataProductCode'    = 'TSSP',
	'extension'          = 'png',
	'locationCode'       = 'CRIP.C1',
	'deviceCategoryCode' = 'CTD',
	'dateFrom'           = '2019-03-20T00:00:00.000Z',
	'dateTo'             = '2019-03-20T00:10:00.000Z',
	'dpo_qualityControl' = '1',
	'dpo_resample'       = 'none'
);
result = onc$orderDataProduct(filters);

onc$print(result)

Returns

A dictionary with a list of download results (one per file) represented as dictionaries, and overall statistics for the process. If files were downloaded, they will appear in the output directory.

{
    "downloadResults": [
        {
            "url": "https://data.oceannetworks.ca/api/dataProductDelivery&in(...)",
            "index": "1",
            "status": "complete",
            "downloaded": true,
            "file": "CRUN_CTDAML_CTD_20190320T000000Z_20190320T003000Z-clean.png",
            "size": 136899,
            "fileDownloadTime": 0.072,
            "requestCount": 16
        },
        (...)
    ],
    "stats": {
        "runTime": 2.374,
        "downloadTime": 0.139,
        "requestCount": 19,
        "totalSize": 249377
    }
}

A struct with a vector of download results (one per file) represented as structs, and overall statistics for the process. If files were downloaded, they will appear in the output directory.

struct(
    downloadResults: [
        struct(
            url: 'https://data.oceannetworks.ca/api/dataProductDelivery?method=(...)'
            status: 'complete'
            size: 142160
            file: 'CRUN_CTDAML_20190320T000000Z_20190320T001000Z-clean.png'
            index: '1'
            downloaded: true
            requestCount: 28
            fileDownloadTime: 0.252
        ),
        (...)
    ]
    stats: struct(
        runTime: 2.633
        downloadTime: 0.709
        requestCount: 30
        totalSize: 266900
    )
)

A named list containing a list of download results (one per file) represented as named lists, and overall statistics for the process. If files were downloaded, they will appear in the output directory.

[
  downloadResults: [
    [
      url: "https://data.oceannetworks.ca/api/dataProductDelivery?method=downl(...)",
      status: "completed",
      size: 142222,
      file: "CRUN_CTDAML_20190320T000000Z_20190320T001000Z-clean.png",
      index: "1",
      downloaded: TRUE,
      requestCount: 30,
      fileDownloadTime: 0.069
    ],
    (...)
  ],
  stats: [
    runTime: 2.49,
    downloadTime: 0.167,
    requestCount: 32,
    totalSize: 267045
  ]
]

Detailed description of returned data [+]

The response is a ^dictionary | struct | named list with two elements: "downloadResults" and "stats".

"downloadResults" contains a list of ^dictionaries | structs | named lists with the download results for each file downloaded, in the following structure:

Property	Type	Description
url	string	The URL used to download the file
index	string	Each data product file has a unique index used to download it. The index can either be a number (starting in 1, increasing onwards) or "meta" for the metadata file.
status	string	The status of the download process for this file. Possible values are: `complete`: The file has been downloaded or the download URL has been returned `error`: The file was not downloaded (or generated) due to an error
downloaded	boolean	Indicates if the file was downloaded
file	string	The filename of the file if it was downloaded (otherwise, an empty string)
size	int	The size of the downloaded file in bytes (0 if it was not downloaded)
fileDownloadTime	float	The time duration (in seconds) spend downloading this file (0 if it was not downloaded)
requestCount	integer	The total count of HTTP requests made to the server to obtain this specific file or its download result

"stats" is a ^dictionary | struct | named list with aggregated statistics on the overall process, with the following structure:

Property	Type	Description
runTime	float	The time duration (in seconds) spend waiting for this data product order to be prepared in the server
downloadTime	float	The time duration (in seconds) spend downloading all files
requestCount	int	The total count of HTTP requests made to the server to order, run and download the full data product
totalSize	int	The total size of all files downloaded (in bytes)

requestDataProduct

Manually request the server to prepare a data product generation process with the filters provided.

When successful, returns the "requestId" that can be used to run the generation process.

The method orderDataProduct() uses this function internally.

Parameter

Type

Description

Example

filters

^filters

A group of filters used to configure the data product request, to be provided to the request method from the API data product delivery service.

The data source is specified by including one of the following sets of filters:

Data source	Filters to include
A device	deviceCode
Property in a device	propertyCode, deviceCode
Property in a location	locationCode, propertyCode
Device category in a location	locationCode, deviceCategoryCode
Property in a device category in a location	locationCode, deviceCategoryCode, propertyCode

Additionally, the following filters are mandatory:

Depending on the data product requested, you must also include its required data product options in the filters. Otherwise, the API will return an error with instructions on what filters to add.

{
'locationCode': 'BACAX',

'deviceCategoryCode': 'ADCP2MHZ',
'dataProductCode': 'TSSD',
'extension': 'csv',
'dateFrom':'2016-07-27T00:00:00.000Z',
'dateTo': '2016-08-01T00:00:00.000Z',
'dpo_qualityControl': 1,
'dpo_resample': 'none',

'dpo_dataGaps': 0
}

{
'locationCode', 'BACAX',

'deviceCategoryCode', 'ADCP2MHZ',
'dataProductCode', 'TSSD',
'extension', 'csv',
'dateFrom','2016-07-27T00:00:00.000Z',
'dateTo', '2016-08-01T00:00:00.000Z',
'dpo_qualityControl', 1,
'dpo_resample', 'none',

'dpo_dataGaps', 0
}

list(
'locationCode' = 'BACAX',

'deviceCategoryCode' = 'ADCP2MHZ',
'dataProductCode' = 'TSSD',
'extension' = 'csv',
'dateFrom'='2016-07-27T00:00:00.000Z',
'dateTo' = '2016-08-01T00:00:00.000Z',
'dpo_qualityControl' = 1,
'dpo_resample' = 'none',

'dpo_dataGaps' = 0
)

(parameters with an underline are required)

Example: Request a data product for a PNG plot

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
	'dataProductCode'   : 'TSSP',
	'extension'         : 'png',
	'locationCode'      : 'CRIP.C1',
	'deviceCategoryCode': 'CTD',
	'dateFrom'          : '2019-03-20T00:00:00.000Z',
	'dateTo'            : '2019-03-20T00:30:00.000Z',
	'dpo_qualityControl': '1',
	'dpo_resample'      : 'none'
}

result = onc.requestDataProduct(filters)

onc.print(result)

onc = Onc(YOUR_TOKEN_HERE);

filters = {
	'dataProductCode',    'TSSP',
	'extension',          'png',
	'locationCode',       'CRIP.C1',
	'deviceCategoryCode', 'CTD',
	'dateFrom',           '2019-03-20T00:00:00.000Z',
	'dateTo',             '2019-03-20T00:10:00.000Z',
	'dpo_qualityControl', '1',
	'dpo_resample',       'none'
};
result = onc.requestDataProduct(filters);

onc.print(result);

library(onc)
onc <- Onc(YOUR_TOKEN_HERE)

filters = list(
	'dataProductCode'    = 'TSSP',
	'extension'          = 'png',
	'locationCode'       = 'CRIP.C1',
	'deviceCategoryCode' = 'CTD',
	'dateFrom'           = '2019-03-20T00:00:00.000Z',
	'dateTo'             = '2019-03-20T00:10:00.000Z',
	'dpo_qualityControl' = '1',
	'dpo_resample'       = 'none'
)
result = onc$requestDataProduct(filters)

onc$print(result)

Returns

A ^dictionary with the information required to run the data product generation (or prepare the download process if the files are found in the archive).

When the requested files already exist in the archive, the response's structure resembles the following example:

{
  "dpRequestId": 3223489,
  "fileSize": 133436160,
  "compressedFileSize": 25142845,
  "numFiles": 4
  "downloadTimes": {
    "10Mbps": 13.343616,
    "50Mbps": 2.668723,
    "150Mbps": 0.8895744
  },
}

struct(
    dpRequestId: 3869210
    fileSize: 122444105
    compressedFileSize: 122444105
    numFiles: 1
    downloadTimes: struct(
        x10Mbps: 12.2444
        x50Mbps: 2.44888
        x150Mbps: 0.816294
    )
)

[
  dpRequestId: 3869211,
  fileSize: 122444105,
  compressedFileSize: 122444105,
  numFiles: 1,
  downloadTimes: [
    10Mbps: 12.2444105,
    50Mbps: 2.448882,
    150Mbps: 0.816294
  ]
]

In contrast, if the data product needs to be generated from scratch, the response's structure resembles the following example:

{
    "dpRequestId": 3223464,
    "estimatedFileSize": "209 kB",
    "estimatedProcessingTime": "20 s",
    "disclaimer": "These are extremely rough to begin with (...)"
}

struct(
    dpRequestId: 3869200
    estimatedFileSize: '209 kB'
    estimatedProcessingTime: '20 s'
    disclaimer: 'These are extremely rough to begin with (...)'
)

[
  dpRequestId: 3869203,
  estimatedFileSize: "209 kB",
  estimatedProcessingTime: "20 s"
  disclaimer: "These are extremely rough to begin with (...)",
]

Note that estimates on size and time for data product generation are still in development and might not be reliable.

Detailed description of returned data [+]

The response contains a ^dictionary with the following structure:

Property	Type	Description	Example
Always contains:
dpRequestId	int	A unique id for a data product request	`3223464`
May contain:
compressedFileSize	int	The compressed size of the data product file(s) in bytes	`12563408`
fileSize	int	The total size of the data product file(s) in bytes	`70766230`
numFiles	int	The number of files to download	`4`
downloadTimes	^dictionary	A ^dictionary of estimated download times for different internet speeds	`{"10Mbps":7.076623,"50Mbps":1.4153247,"150Mbps":0.47177488}`
estimatedFileSize	string	The estimated file size of the generated data product	`209 kB`
estimatedProcessingTime	string	The estimated time, in seconds, that it will take to run the data product request	`16 s`
disclaimer	string	A message from our development team	"(...)"

runDataProduct

Manually request to run an existing data product request with the provided "requestId".

When successful, returns the "runId" that can be used to download the data product files. This method also prints progress messages to the console.

The method orderDataProduct() uses this function internally.

Parameter

Type

Description

Example

requestId

int

The "dpRequestId" request identifier returned by the requestDataProduct() method, to be provided to the run method in the API data product delivery service.

2046404

waitComplete

boolean

Whether this method should wait for the server to finish the data product preparation before continuing.

^False: Only invoke the run operation for the data product request
^True: Invoke the run operation, and poll the server until the response confirms that the files are ready to be downloaded

Default value: ^True

^True

(parameters with an underline are required)

Example: Run a data product with a request id

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

result = onc.runDataProduct(3223464, waitComplete=True)

onc.print(result)

onc = Onc(YOUR_TOKEN_HERE);

result = onc.runDataProduct(3223464, 'waitComplete', true);

onc.print(result);

library(onc)
onc <- Onc(YOUR_TOKEN_HERE)

result = onc$runDataProduct(3223464, waitComplete = TRUE)

onc$print(result)

Returns

A dictionary with a list of "runIds" used to download the files and additional run information, with a structure as in the following example:

{
    "runIds": [9505160],
    "fileCount": 2,
    "runTime": 1.469635009765625,
    "requestCount": 1
}

A struct with a vector (can be of length 1) of "runIds" used to download the files and additional run information, as in the following example:

struct(
    runIds: 11328534
    fileCount: 1
    runTime: 2.154
    requestCount: 4
)

A named list that includes a list of "runIds" used to download the files and additional run information, as in the following example:

[
  runIds: [11337039],
  fileCount: 2,
  runTime: 16.138,
  requestCount: 5
]

The "fileCount" is only reliable if the method was invoked with waitComplete=True

Detailed description of returned data [+]

The response is a ^dictionary with the following properties:

Property	Type	Description	Example
runIds	list	A list of integers. The run ids obtained for this data product run. The ids can be provided to the `downloadDataProduct()` method	`[ 9505160 ]`
fileCount	int	The number of files available for download. Only reliable if the method was invoked with `waitComplete=True`	`2`
runTime	float	Time (in seconds) spend on this operation	`1.469`
requestCount	int	The total count of HTTP requests made to the server during this operation	`1`

downloadDataProduct

Manually download all files for a data product file with the provided "runId".

When successful, downloads all files and returns information on the download process. Will print progress messages to the console.

The method orderDataProduct() uses this function internally.

Parameter	Type	Description	Example
runId	int	The "`dpRunId`" run identifier returned by the `runDataProduct()` method, to be provided to the download method in the API data product delivery service.	`9505160`
maxRetries	int	The number of times to poll the service asking if the product is ready for download, before the method aborts. Default: 0 (no limit).	`1000`
downloadResultsOnly	boolean	Whether the files will be downloaded or if only the download URL for each file will be returned. ^True: Files will not be downloaded. The file download URLs will be included in the method results ^False: Files are downloaded to the output path Default: ^False	`^False`
includeMetadataFile	boolean	Indicates if the metadata file associated with the data product request will be downloaded. ^True: Metadata file will be downloaded ^False: Metadata file will not be downloaded Default: ^True	`^True`
overwrite	boolean	Whether new files downloaded will overwrite previous files with the same filename found in the output directory. ^True: Overwrite files with the same filename ^False: Do not overwrite files (downloaded files with the same filename will be skipped) Default: ^False	`^False`

(parameters with an underline are required)

Example: Download all data product files with a run id

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

result = onc.downloadDataProduct(9505160)

onc.print(result)

onc = Onc(YOUR_TOKEN_HERE);

result = onc.downloadDataProduct(9505160);

onc.print(result);

library(onc)
onc <- Onc(YOUR_TOKEN_HERE)

result = onc$downloadDataProduct(9505160)

onc$print(result)

Returns

A list of dictionaries with download results for every file downloaded.

[
    {
        "url": "https://data.oceannetworks.ca/api/dataProductDelivery?(...)",
        "status": "complete",
        "size": 132719,
        "file": "CRUN_CTDAML_CTD_20190320T000000Z_20190321T000000Z-clean.png",
        "index": "1",
        "downloaded": true,
        "requestCount": 1,
        "fileDownloadTime": 0.088
    },
    (...)
]

A list of structs with download results for every file downloaded.

[
    struct(
        url: 'https://data.oceannetworks.ca/api/dataProductDelivery?(...)'
        status: 'complete'
        size: 164255
        file: 'CRUN_CTDAML_CTD_20190320T000000Z_20190321T000000Z-clean.png'
        index: '1'
        downloaded: true
        requestCount: 1
        fileDownloadTime: 0.24
    ),
    (...)
]

A list of named lists with download results for every file downloaded for this data product.

[
  [
    url: "https://data.oceannetworks.ca/api/dataProductDelivery?(...)",
    status: "complete",
    size: 142228,
    file: "CRUN_CTDAML_CTD_20190320T000000Z_20190321T000000Z-clean.png",
    index: "1",
    downloaded: TRUE,
    requestCount: 1,
    fileDownloadTime: 0.064
  ],
  (...)
]

Detailed description of returned data [+]

Returns a list of ^dictionaries with the download information for each file downloaded, with the following properties:

Property	Type	Description
url	string	The URL used to download the file
index	string	Each data product file has a unique index used to download it. The index can either be a number (starting in 1, increasing onwards) or "meta" for the metadata file.
status	string	The status of the download process for this file. Possible values are: `complete`: The file has been downloaded or the download URL has been returned `error`: The file was not downloaded (or generated) due to an error
downloaded	boolean	Indicates if the file was downloaded
file	string	The filename of the file if it was downloaded (otherwise, an empty string)
size	int	The size of the downloaded file in bytes (0 if it was not downloaded)
fileDownloadTime	float	The time duration (in seconds) spend downloading this file (0 if it was not downloaded)
requestCount	integer	The total count of HTTP requests made to the server to obtain this specific file or its download result

Page tree

Data product download methods

Summary

orderDataProduct

requestDataProduct

Returns

runDataProduct

Returns

downloadDataProduct

Returns