Page tree
Skip to end of metadata
Go to start of metadata

Data product download methods allow you to order and download more than 120 different types of ONC data products. They provide granular control over what data to obtain, from where, and in what specific time frame.

These methods provide your scripts the download functionality from ONC's Data Search tool. Examples of usage include:

  • Downloading PNG plots of the sensor readings in a device
  • Downloading ADCP readings as text files or in specific manufacturer formats
  • Downloading compressed or raw audio files from hydrophones


Data product download methods allow you to request any kind of available data product at any timeframe with your own configuration; if it doesn't exist in our archive, the product will be automatically generated before your download starts.

Summary

Method

Description


orderDataProduct (filters, maxRetries, downloadResultsOnly, includeMetadataFile)

Request, run and download a data productlink

requestDataProduct (filters)

Request a data productlink

runDataProduct (dpRequestId, waitComplete)

Run a data product requestlink

downloadDataProduct (runId, maxRetries, downloadResultsOnly, includeMetadataFile, overwrite)

Download an available data productlink



orderDataProduct 

Request, run and download a data product and downloads the generated files to the outPath directory as specified in the ONC class constructor ("output" by default). The download directory will be created automatically if it doesn't exist.

This method executes the complete data product download workflow described in the API Data Product Delivery Service and will usually suffice for downloading data products.

Parameter

Type

Description

Example

filtersdictionary

A dictionary of filters used to configure the data product request, to be provided to the request method from the API data product delivery service.

The data source is specified by including one of the following sets of filters:

Data sourceFilters to include
A devicedeviceCode
Property in a device

propertyCodedeviceCode

Property in a location

locationCodepropertyCode

Device category in a location

locationCodedeviceCategoryCode

Property in a device category in a location

locationCodedeviceCategoryCodepropertyCode

Additionally, the following filters are mandatory:

Depending on the data product requested, you must also include its required data product options in the filters. Otherwise, the API will return an error with instructions on what filters to add.

{
'locationCode':'BACAX',

'deviceCategoryCode':'ADCP2MHZ',
'dataProductCode':'TSSD',
'extension':'csv',
'dateFrom':'2016-07-27T00:00:00.000Z',
'dateTo':'2016-08-01T00:00:00.000Z',
'dpo_qualityControl':1,
'dpo_resample':'none',
'dpo_dataGaps':0
}

maxRetriesint

The number of times to poll the service asking if the product is ready for download, before the method aborts.

Default: 0 (no limit).

1000
downloadResultsOnlyboolean

Whether the files will be downloaded or if only the download URL for each file will be returned.

  • True: Files will not be downloaded. The file download URLs will be included in the method results
  • False: Files are downloaded to the output path

Default: False

True
includeMetadataFileboolean

Indicates if the metadata file associated with the data product request will be downloaded.

  • True: Metadata file will be downloaded
  • False: Metadata file will not be downloaded

Default: False

True

overwriteboolean

Whether new files downloaded will overwrite previous files with the same filename found in the output directory.

  • True: Overwrite files with the same filename
  • False: Do not overwrite files (downloaded files with the same filename will be skipped)

Default: False

False

(parameters with an underline are required)

Example: Download a PNG plot for all properties in a device category in a location
from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
	'dataProductCode'   : 'TSSP',
	'extension'         : 'png',
	'locationCode'      : 'CRIP.C1',
	'deviceCategoryCode': 'CTD',
	'dateFrom'          : '2019-03-20T00:00:00.000Z',
	'dateTo'            : '2019-03-20T00:30:00.000Z',
	'dpo_qualityControl': '1',
	'dpo_resample'      : 'none'
}

result = onc.orderDataProduct(filters)
onc.print(result)

Returns

A dictionary with a list of download results (one per file) represented as dictionaries, and overall statistics for the process. If files were downloaded, they will appear in the output directory.

Example of returned download results
{
    "downloadResults": [
        {
            "url": "https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=YOUR_TOKEN&index=1",
            "index": "1",
            "status": "complete",
            "downloaded": true,
            "file": "CampbellRiverUnderwaterNetwork_CTDAML_CTD_20190320T000000Z_20190320T003000Z-clean.png",
            "size": 136899,
            "fileDownloadTime": 0.072,
            "requestCount": 16
        }
    ],
    "stats": {
        "runTime": 2.374,
        "downloadTime": 0.139,
        "requestCount": 19,
        "totalSize": 249377
    }
}
 Detailed description of returned data [+]

The response is a dictionary with two keys: "downloadResults" and "stats".

"downloadResults" contains a list of dictionaries with the download information for each file in the data product, with the following keys:

Property

Type

Description

urlstringThe URL used to download the file
indexstringEach data product file has a unique index used to download it. The index can either be a number (starting in 1, increasing onwards) or "meta" for the metadata file.
statusstring

The status of the download process for this file. Possible values are:

  • complete: The file has been downloaded or the download URL has been returned
  • error: The file was not downloaded (or generated) due to an error
downloadedbooleanIndicates if the file was downloaded
filestringThe filename of the file if it was downloaded (otherwise, an empty string)
sizeintThe size of the downloaded file in bytes (0 if it was not downloaded)
fileDownloadTimefloatThe time duration (in seconds) spend downloading this file (0 if it was not downloaded)
requestCountintegerThe total count of HTTP requests made to the server to obtain this specific file or its download result

"stats" is a dictionary with information of the overall process, with the following keys:

Property

Type

Description

runTimefloatThe time duration (in seconds) spend waiting for this data product order to be prepared in the server
downloadTimefloatThe time duration (in seconds) spend downloading all files
requestCountintThe total count of HTTP requests made to the server to order, run and download the full data product
totalSizeintThe total size of all files downloaded (in bytes)



requestDataProduct

Manually request the server to prepare a data product generation with the filters provided. When successful, returns the "requestId" that can be used to run the generation process.

The method orderDataProduct() uses this function internally when requesting and downloading data products.

Parameter

Type

Description

Example

filtersdictionary

A dictionary of filters used to configure the data product request, to be provided to the request method from the API data product delivery service.

The data source is specified by including one of the following sets of filters:

Data sourceFilters to include
A devicedeviceCode
Property in a device

propertyCodedeviceCode

Property in a location

locationCodepropertyCode

Device category in a location

locationCodedeviceCategoryCode

Property in a device category in a location

locationCodedeviceCategoryCodepropertyCode

Additionally, the following filters are mandatory:

Depending on the data product requested, you must also include its required data product options in the filters. Otherwise, the API will return an error with instructions on what filters to add.

{
'locationCode':'BACAX',

'deviceCategoryCode':'ADCP2MHZ',
'dataProductCode':'TSSD',
'extension':'csv',
'dateFrom':'2016-07-27T00:00:00.000Z',
'dateTo':'2016-08-01T00:00:00.000Z',
'dpo_qualityControl':1,
'dpo_resample':'none',
'dpo_dataGaps':0
}

(parameters with an underline are required)

Example: Request a data product for a PNG plot
from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
	'dataProductCode'   : 'TSSP',
	'extension'         : 'png',
	'locationCode'      : 'CRIP.C1',
	'deviceCategoryCode': 'CTD',
	'dateFrom'          : '2019-03-20T00:00:00.000Z',
	'dateTo'            : '2019-03-20T00:30:00.000Z',
	'dpo_qualityControl': '1',
	'dpo_resample'      : 'none'
}

result = onc.requestDataProduct(filters)
onc.print(result)

Returns

A dictionary with the information required to run the data product generation (or prepare the download process if the files are found in the archive).

When the files are found in the archive, the response has the structure described in the following example:

Example response when files are found in the archive
{
  "compressedFileSize": 25142845,
  "downloadTimes": {
    "10Mbps": 13.343616,
    "50Mbps": 2.668723,
    "150Mbps": 0.8895744
  },
  "dpRequestId": 3223489,
  "fileSize": 133436160,
  "numFiles": 4
}

In contrast, when the data product will be generated from scratch, the response is as follows:

Example response for a data product that will be generated
{
    "dpRequestId": 3223464,
    "estimatedFileSize": "209 kB",
    "estimatedProcessingTime": "20 s",
    "disclaimer": "These are extremely rough to begin with (...)"
}

Note that estimates on size and time for data product generation are still in development and might not be reliable.

 Detailed description of returned data [+]

The response is a dictionary with the following keys:

Property

Type

Description

Example
Always contains:
dpRequestIdintA unique id for a data product request

3223464

May contain:
compressedFileSizeintThe compressed size of the data product file(s) in bytes12563408
fileSizeintThe total size of the data product file(s) in bytes70766230
numFilesintThe number of files to download4
downloadTimesdictA dictionary of estimated download times for different internet speeds{"10Mbps":7.076623,"50Mbps":1.4153247,"150Mbps":0.47177488}
estimatedFileSizestringThe estimated file size of the generated data product209 kB
estimatedProcessingTimestringThe estimated time, in seconds, that it will take to run the data product request16 s
disclaimerstringA message from our development team"(...)"



runDataProduct

Manually request the server to run a data product request with the provided "requestId". When successful, returns the "runId" that can be used to download the data product files. The method also prints messages to the console to keep track of progress.

The method orderDataProduct() uses this function internally when requesting and downloading data products.

Parameter

Type

Description

Example

requestIdint

The "dpRequestId" request identifier returned by the requestDataProduct() method, to be provided to the run method in the API data product delivery service.

2046404

waitComplete

boolean

Whether this method should wait for the server to finish the data product preparation before continuing.

  • False: Only invoke the run operation for the data product request
  • True: Invoke the run operation, and poll the server until the response confirms that the files are ready to be downloaded

Default value: True

True

(parameters with an underline are required)

Example: Run a data product with a request id
from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

result = onc.runDataProduct(3223464, waitComplete=True)
onc.print(result)

Returns

A dictionary with a list of "runIds" required to download the data product files, and other information obtained during the run process.

The response has the structure described in the following example:

Example response for a run request
{
    "runIds": [9505160],
    "fileCount": 2,
    "runTime": 1.469635009765625,
    "requestCount": 1
}

The "fileCount" is only reliable if the method was invoked with waitComplete=True

 Detailed description of returned data [+]

The response is a dictionary with the following keys:

Property

Type

Description

Example
runIdslistA list of integers. The run ids obtained for this data product run. The ids can be provided to the downloadDataProduct() method

[ 9505160 ]

fileCountint

The number of files available for download.

Only reliable if the method was invoked with waitComplete=True

2
runTimefloatTime (in seconds) spend on this operation1.469
requestCountintThe total count of HTTP requests made to the server during this operation1



downloadDataProduct

Manually download all files for a data product file with the provided "runId". When successful, downloads all files and returns information on the download process. Will print messages to the console to keep track of progress.

The method orderDataProduct() uses this function internally when requesting and downloading data products.

Parameter

Type

Description

Example

runIdint

The "dpRunId" run identifier returned by the runDataProduct() method, to be provided to the download method in the API data product delivery service.

9505160

maxRetriesint

The number of times to poll the service asking if the product is ready for download, before the method aborts.

Default: 0 (no limit).

1000
downloadResultsOnlyboolean

Whether the files will be downloaded or if only the download URL for each file will be returned.

  • True: Files will not be downloaded. The file download URLs will be included in the method results
  • False: Files are downloaded to the output path

Default: False

True
includeMetadataFileboolean

Indicates if the metadata file associated with the data product request will be downloaded.

  • True: Metadata file will be downloaded
  • False: Metadata file will not be downloaded

Default: True

True

overwriteboolean

Whether new files downloaded will overwrite previous files with the same filename found in the output directory.

  • True: Overwrite files with the same filename
  • False: Do not overwrite files (downloaded files with the same filename will be skipped)

Default: False

False

(parameters with an underline are required)

Example: Download all data product files with a run id
from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

result = onc.downloadDataProduct(9505160)
onc.print(result)

Returns

A list of dictionaries with download results for every file downloaded for this data product.

Example response with download information for a file
[
    {
        "url": "https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=YOUR_TOKEN&index=1",
        "status": "complete",
        "size": 132719,
        "file": "CampbellRiverUnderwaterNetwork_CTDAML_CTD_20190320T000000Z_20190321T000000Z-clean.png",
        "index": "1",
        "downloaded": true,
        "requestCount": 1,
        "fileDownloadTime": 0.088
    }
]
 Detailed description of returned data [+]

Returns a list of dictionaries with the download information for each file in the data product, with the following keys:

Property

Type

Description

urlstringThe URL used to download the file
indexstringEach data product file has a unique index used to download it. The index can either be a number (starting in 1, increasing onwards) or "meta" for the metadata file.
statusstring

The status of the download process for this file. Possible values are:

  • complete: The file has been downloaded or the download URL has been returned
  • error: The file was not downloaded (or generated) due to an error
downloadedbooleanIndicates if the file was downloaded
filestringThe filename of the file if it was downloaded (otherwise, an empty string)
sizeintThe size of the downloaded file in bytes (0 if it was not downloaded)
fileDownloadTimefloatThe time duration (in seconds) spend downloading this file (0 if it was not downloaded)
requestCountintegerThe total count of HTTP requests made to the server to obtain this specific file or its download result
  • No labels