The API dataProductDelivery service provides the ability to request, run and download Data Products for the following:
Downloading a data product is done with a three step process:
1. Request a data product creation task to be defined using the request method
Validates request parameters, creates a new request definition and provides performance estimates
2. Run the data product request using the run method
Starts the data product generation process by adding the request to the Task Queue
3. Download the data product using the download method
Attempts to download a file at an index. If the file at that index has not been available for download at the time of the download request, the service will return information about the status of the process. The client application will need to continue to request the download until the file is ready to be downloaded, or an error is encountered.
http://data.oceannetworks.ca/api/dataProductDelivery |
Method | Description | Example |
---|---|---|
request | Retrieve raw data available at given station. |
|
run | Run the search generated by request |
|
download | Download one result file for a data product. |
|
The request method creates a search and returns information regarding the number of files, file size, compressed file size, and estimated download times as well as a request id that can be used to run the data product using the run method.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
token | string | All Web Services require a token. This can be generated at http://dmas.uvic.ca/Profile. Click on the "Web Services" tab and click "Generate Token" | token=5ba514e8-1203-428c-8835-879c8173e387 |
dataProductCode | string | Request a Data Product for a specific Data Product Code
| dataProductCode=TSSD |
extension | string | Request a Data Product for a specific File Format Extension
| extension=csv |
begin | datetime | Request a Data Product for data starting at a specific date/time Date Time format: yyyy-MM-dd'T'HH:mm:ss'.'SSS'Z'
| begin=2016-07-25T00:00:00.000Z |
end | datetime | Request a Data Product for data ending at a specific date/time Date Time format: yyyy-MM-dd'T'HH:mm:ss'.'SSS'Z'
| end=2016-07-29T00:00:00.000Z |
Conditionally Required | |||
locationCode | string | Request a Data Product for data from a specific Location
| locationCode=BACAX |
deviceCategoryCode | string | Request a Data Product for data from devices belonging to a specific Device Category
| deviceCategoryCode=ADCP2MHZ |
deviceCode | string | Request a Data Product for data from a specific Device
| deviceCode=AandOpt0581 |
propertyCode | string | Request a Data Product for data for a specific Property
| property=pressure |
Data Product Options | string | Each Data Product Extension may have required data product options that determine how to process or package the data.
| dpo_qualityControl=1 dpo_resample=average dpo_average=600 |
Search Type Equivalent | Parameters Given |
---|---|
Instrument by location (Device Level) | locationCode, deviceCategoryCode |
Instrument by location (Sensor Level) | locationCode, deviceCategoryCode, propertyCode |
Instrument by Category (Device Level) | deviceCode |
Instrument by Category (Sensor Level) | deviceCode, propertyCode |
Variable by Location | locationCode, propertyCode |
Detailed information will be returned if the files are in archive. If the data product needs to be generated, only a dpRequestId will be returned.
{ "compressedFileSize" :12563408, "downloadTimes" :{ "10Mbps" :7.076623, "50Mbps" :1.4153247, "150Mbps" :0.47177488}, "dpRequestId" :1852695, "fileSize" :70766230, "numFiles" :4} |
Property | Type | Description | Example |
---|---|---|---|
dpRequestId | integer | Returns a numerical request id to be passed into the "run" method (SearchHdrId) | "dpRequestId":1852695 |
numFiles | integer | Returns the number of files that meet the criteria of the search | "numFiles":4 |
fileSize | Long | The file size (in bytes) | "fileSize":70766230 |
compressedFileSize | Long | The file size (in bytes) after compression | "compressedFileSize":12563408 |
downloadTimes | String':'Float | A set of three common download speeds and the time (in seconds) it would take to download the files for each speed | "downloadTimes":{"10Mbps":7.076623,"50Mbps":1.4153247,"150Mbps":0.47177488} |
{ "dpRequestId" :1908432, "estimatedFileSize" : "11940760" , "estimatedProcessingTime" : "21" } |
Property | Type | Description | Example |
---|---|---|---|
dpRequestId | integer | Returns a numerical request id to be passed into the "run" method (SearchHdrId) | "dpRequestId":1908432 |
estimatedFileSize | string | The estimated file size (in bytes) | "estimatedFileSize":"11940760" or "estimatedFileSize":"no estimated file size avalible." |
estimatedProcessingTime | string | The estimated time it will take to process the data proeuct | "estimatedProcessingTime":"21" or "estimatedProcessingTime":"No processing time estimates found for this type of data." |
errorCode | errorMessage | Description |
---|---|---|
25 | Invalid Time Range, Start Time is in the future. | Occurs when the end is before the begin date/time.
|
127 | Invalid parameter value | Occurs when an invalid code is used in the filter. Most filters require an exact match, otherwise this error will occur.
|
128 | Missing parameter | Occurs when multiple parameters are needed, but not all are present. Occurs when begin is used without end
|
128 | Requested type can not be determined | Occurs when the one of the 5 possible parameter combinations is not used.
"possible parameter combinations are: locationCode, deviceCategoryCode; locationCode, deviceCategoryCode, propertyCode; locationCode, propertyCode; deviceCode; deviceCode, propertyCode" |
129 | Invalid parameter name | Occurs when a filter parameter is used, but is not supported. |
The run method runs the data product created by a call to the request method.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
token | string | All Web Services require a token. This can be generated at http://dmas.uvic.ca/Profile. Click on the "Web Services" tab and click "Generate Token" | token=5ba514e8-1203-428c-8835-879c8173e387 |
dpRequestId | integer | A dpRequestId returned from the request method. (SearchHdrId) | dpRequestId=1852695 |
HTTP status code 200 is returned when all of the runs in the data product delivery request have completed running.
[ { "dpRunId": 4521400, "status": "complete" } ] |
The payload contains a list of runs, each run contains the following properties
Property | Type | Description | Example |
---|---|---|---|
dpRunId | integer | A Run Id that can be used to download the data product when processing is complete. (AKA, SearchId) | "dpRunId":4521400 |
status | string | The current status of data product request. Valid values are ("cancelled", "queued", "error", "running" and "complete") | "status": "complete" |
HTTP status code 202 is returned if all of the runs in the data product delivery request have not completed running.
[ { "dpRunId": 4521404, "status": "Running" } ] |
The payload contains a list of runs, each run can contain the following properties
Property | Type | Description | Example |
---|---|---|---|
dpRunId | integer | A Run Id that can be used to download the data product when processing is complete. (AKA, SearchId) | "dpRunId":4521400 |
status | string | The current status of data product request. Valid values are ("queued", "error" and "running") | "status": "complete" |
queuePosition | An Integer | only returned if status is "queued". it starts at 1 meaning it is the first item in the queue | "queuePosition": 10 |
message | string | An error message is returned when the status is "error" | "message": "Unable to calculate position in queue." |
{ "errors": [ { "errorCode": 127, "errorMessage": "Invalid parameter value", "parameter": "token" } ] } |
errorCode | errorMessage | Description |
---|---|---|
127 | Invalid parameter value | Occurs when an invalid token is used. |
128 | Missing parameter | Occurs when the token parameter is not included in the request |
{ "errors": [ { "errorCode": 129, "errorMessage": "Invalid parameter name", "parameter": "RequestId" }, { "errorCode": 128, "errorMessage": "Missing parameter", "parameter": "dpRequestId" } ] } |
errorCode | errorMessage | Description |
---|---|---|
127 | Invalid search header ID. | Occurs when an invalid Request ID is used in the dpRequestId paramters. |
128 | Missing parameter | Occurs when multiple parameters are needed, but not all are present. Occurs when begin is used without end
|
129 | Invalid parameter name | Occurs when a filter parameter is used, but is not supported. |
The download method downloads a file for the specified data product search. The file to download is specified by index with the first valid index being 1 and the last being the total number of files generated by the search.
Parameter | Type | Description | Example |
---|---|---|---|
Required | |||
token | string | All Web Services require a token. This can be generated at http://dmas.uvic.ca/Profile. Click on the "Web Services" tab and click "Generate Token" | token=5ba514e8-1203-428c-8835-879c8173e387 |
dpRunId | integer | The dpRunId returned from the run method. (SearchId) | |
index | integer | The index of the file to be downloaded, valid values are 1 to the number of result files. If the index is greater than the number of result files a response code of 204, indicating no file at that index is returned. |
HTTP status code 200 will be returned in HTTP header and the requested file will be downloaded.
Response Code | Description |
---|---|
200 - Ok | The file is included in the payload. |
202 - Accepted | The data product and index are valid but the data product is still being processed. This status can be used to poll waiting for data product completion. The payload includes the comment as displayed in the GUI. |
204 - No Content | There is no result file for the data product at the index specified. This can be used to stop looping through result files when the actual number of result files isn't known. |
400 - Bad Request | Missing required parameters |
dpRunId is not a number | |
index is not a positive number or not meta | |
dpRunId is not a valid data product run Id | |
The data product selected by dpRunId was run by a different user | |
Index is meta but the data product does not include metadata | |
401 - Unauthorized | Token is invalid |
404 - Not Found | Index number is greater than the number of files |
410 - Gone | The result file selected by dpRunId and index is not in expected location and may have been removed during FTP clean up. |
500 - Internal Server Error | This is an unhandled exception and should not occur. All 500 errors should be immediately reported and a JIRA ticket created. |
The following is an example of the return of data product that is still being processed, note that the response code in this case is 202.
{ "message": "Transferring (BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv) to the FTP server", "status": "running" } |
Property | Type | Description | Example |
---|---|---|---|
status | string | The current status of data product request. Valid values are ("queued", "error" and "running") | "status": "running" |
message | string | The latest status message reported by the data product process. | "message": "Transferring (BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv) to the FTP server" |