Data product download methods allow you to order and download more than 120 different types of ONC data products. They provide granular control over what data to obtain, from where, and in what specific time frame.
These methods provide your scripts the download functionality from ONC's Data Search tool. Examples of usage include:
- Downloading PNG plots of the sensor readings in a device
- Downloading ADCP readings as text files or in specific manufacturer formats
- Downloading compressed or raw audio files from hydrophones
Data product download methods allow you to request any kind of available data product at any timeframe with your own configuration; if it doesn't exist in our archive, the product will be automatically generated before your download starts.
Summary
Method | Description | |
---|---|---|
| Request, run and download a data product | link |
| Request a data product | link |
| Run a data product request | link |
| Download an available data product | link |
orderDataProduct
Request, run and download a data product and downloads the generated files to the outPath
directory as specified in the ONC class constructor ("output" by default). The download directory will be created automatically if it doesn't exist.
This method executes the complete data product download workflow described in the API Data Product Delivery Service and will usually suffice for downloading data products.
Parameter | Type | Description | Example | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
filters | dictionary | A dictionary of filters used to configure the data product request, to be provided to the request method from the API data product delivery service. The data source is specified by including one of the following sets of filters:
Additionally, the following filters are mandatory: Depending on the data product requested, you must also include its required data product options in the filters. Otherwise, the API will return an error with instructions on what filters to add. |
| ||||||||||||
maxRetries | int | The number of times to poll the service asking if the product is ready for download, before the method aborts. Default: 0 (no limit). | 1000 | ||||||||||||
downloadResultsOnly | boolean | Whether the files will be downloaded or if only the download URL for each file will be returned.
Default: False | True | ||||||||||||
includeMetadataFile | boolean | Indicates if the metadata file associated with the data product request will be downloaded.
Default: False |
| ||||||||||||
overwrite | boolean | Whether new files downloaded will overwrite previous files with the same filename found in the output directory.
Default: False | False |
(parameters with an underline are required)
Example: Download a PNG plot for all properties in a device category in a location from onc.onc import ONC onc = ONC('YOUR_TOKEN_HERE') filters = { 'dataProductCode' : 'TSSP', 'extension' : 'png', 'locationCode' : 'CRIP.C1', 'deviceCategoryCode': 'CTD', 'dateFrom' : '2019-03-20T00:00:00.000Z', 'dateTo' : '2019-03-20T00:30:00.000Z', 'dpo_qualityControl': '1', 'dpo_resample' : 'none' } result = onc.orderDataProduct(filters) onc.print(result) |
Returns
A dictionary with a list of download results (one per file) represented as dictionaries, and overall statistics for the process. If files were downloaded, they will appear in the output directory. Example of returned download results { "downloadResults": [ { "url": "https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=YOUR_TOKEN&index=1", "index": "1", "status": "complete", "downloaded": true, "file": "CampbellRiverUnderwaterNetwork_CTDAML_CTD_20190320T000000Z_20190320T003000Z-clean.png", "size": 136899, "fileDownloadTime": 0.072, "requestCount": 16 } ], "stats": { "runTime": 2.374, "downloadTime": 0.139, "requestCount": 19, "totalSize": 249377 } } |
requestDataProduct
Manually request the server to prepare a data product generation with the filters provided. When successful, returns the "requestId"
that can be used to run the generation process.
The method orderDataProduct()
uses this function internally when requesting and downloading data products.
Parameter | Type | Description | Example | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
filters | dictionary | A dictionary of filters used to configure the data product request, to be provided to the request method from the API data product delivery service. The data source is specified by including one of the following sets of filters:
Additionally, the following filters are mandatory: Depending on the data product requested, you must also include its required data product options in the filters. Otherwise, the API will return an error with instructions on what filters to add. |
|
(parameters with an underline are required)
Example: Request a data product for a PNG plot from onc.onc import ONC onc = ONC('YOUR_TOKEN_HERE') filters = { 'dataProductCode' : 'TSSP', 'extension' : 'png', 'locationCode' : 'CRIP.C1', 'deviceCategoryCode': 'CTD', 'dateFrom' : '2019-03-20T00:00:00.000Z', 'dateTo' : '2019-03-20T00:30:00.000Z', 'dpo_qualityControl': '1', 'dpo_resample' : 'none' } result = onc.requestDataProduct(filters) onc.print(result) |
Returns
A dictionary with the information required to run the data product generation (or prepare the download process if the files are found in the archive). When the files are found in the archive, the response has the structure described in the following example: Example response when files are found in the archive { "compressedFileSize": 25142845, "downloadTimes": { "10Mbps": 13.343616, "50Mbps": 2.668723, "150Mbps": 0.8895744 }, "dpRequestId": 3223489, "fileSize": 133436160, "numFiles": 4 } In contrast, when the data product will be generated from scratch, the response is as follows: Example response for a data product that will be generated { "dpRequestId": 3223464, "estimatedFileSize": "209 kB", "estimatedProcessingTime": "20 s", "disclaimer": "These are extremely rough to begin with (...)" } Note that estimates on size and time for data product generation are still in development and might not be reliable. |
runDataProduct
Manually request the server to run a data product request with the provided "requestId"
. When successful, returns the "runId
" that can be used to download the data product files. The method also prints messages to the console to keep track of progress.
The method orderDataProduct()
uses this function internally when requesting and downloading data products.
Parameter | Type | Description | Example |
---|---|---|---|
requestId | int | The " |
|
waitComplete | boolean | Whether this method should wait for the server to finish the data product preparation before continuing.
Default value: True | True |
(parameters with an underline are required)
Example: Run a data product with a request id from onc.onc import ONC onc = ONC('YOUR_TOKEN_HERE') result = onc.runDataProduct(3223464, waitComplete=True) onc.print(result) |
Returns
A dictionary with a list of " The response has the structure described in the following example: Example response for a run request { "runIds": [9505160], "fileCount": 2, "runTime": 1.469635009765625, "requestCount": 1 } The " |
downloadDataProduct
Manually download all files for a data product file with the provided "runId
". When successful, downloads all files and returns information on the download process. Will print messages to the console to keep track of progress.
The method orderDataProduct()
uses this function internally when requesting and downloading data products.
Parameter | Type | Description | Example |
---|---|---|---|
runId | int | The " |
|
maxRetries | int | The number of times to poll the service asking if the product is ready for download, before the method aborts. Default: 0 (no limit). | 1000 |
downloadResultsOnly | boolean | Whether the files will be downloaded or if only the download URL for each file will be returned.
Default: False | True |
includeMetadataFile | boolean | Indicates if the metadata file associated with the data product request will be downloaded.
Default: False |
|
overwrite | boolean | Whether new files downloaded will overwrite previous files with the same filename found in the output directory.
Default: False | False |
(parameters with an underline are required)
Example: Download all data product files with a run id from onc.onc import ONC onc = ONC('YOUR_TOKEN_HERE') result = onc.downloadDataProduct(3223464) onc.print(result) |
Returns
A list of dictionaries with download results for every file downloaded for this data product. Example response with download information for a file [ { "url": "https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=YOUR_TOKEN&index=1", "status": "complete", "size": 132719, "file": "CampbellRiverUnderwaterNetwork_CTDAML_CTD_20190320T000000Z_20190321T000000Z-clean.png", "index": "1", "downloaded": true, "requestCount": 1, "fileDownloadTime": 0.088 } ] |