Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 13 Next »

Description

The API dataProductDelivery service provides the ability to request, run and download Data Products for the following:

  1. Locations & Device Categories
  2. Locations, Device Categories & Properties
  3. Locations & Properties
  4. Devices
  5. Devices & Properties

 

Downloading a data product is done with a three-step process:

1. Request a data product creation task using the request method

Validates request parameters, creates a new request definition and provides performance estimates

    • Use filters to supply the Who, What, Where, When and How of the data product request
      • Who: Token
      • What: Device Category and/or Property
      • Where: Location or Device
      • When: Begin and End date/time
      • How: Data Product, extension and Data Product Options

2. Run the data product request using the run method

Starts the data product generation process by adding the request to the Task Queue 

 

3. Download the data product using the download method

Attempts to download a file at an index. If the file at that index is not ready for download at the time of the download request, the service returns information about the status of the process. The client application must continue to request the download until the file is ready, or an error is encountered.

URL

http://data.oceannetworks.ca/api/dataProductDelivery

Method

Description

Example

request

Create a request for a data product from a filter criteria parameters.

method=request

runRun a data product created by the request method.

method=run

downloadDownload one of the data product result files by index number.

method=download

request

The request method creates a search and returns information regarding the number of files, file size, compressed file size, and estimated download times as well as a request id that can be used to run the data product using the run method. 

Parameters

ParameterTypeDescriptionExample
Required
tokenstringAll Web Services require a token. This can be generated at http://dmas.uvic.ca/Profile. Click on the "Web Services" tab and click "Generate Token"token=5ba514e8-1203-428c-8835-879c8173e387
dataProductCodestring

Request a Data Product for a specific Data Product Code

dataProductCode=TSSD
extensionstring

Request a Data Product for a specific File Format Extension

  • Extension must be valid and match exactly, including case
  • Extensions available for specific Data Product Codes can be obtained programatically from the dataProducts service or interactively from the Available Data Products page
extension=csv
begindatetime

Request a Data Product for data starting at a specific date/time

Date Time format: yyyy-MM-dd'T'HH:mm:ss'.'SSS'Z'

begin=2016-07-25T00:00:00.000Z
enddatetime

Request a Data Product for data ending at a specific date/time

Date Time format: yyyy-MM-dd'T'HH:mm:ss'.'SSS'Z'

end=2016-07-29T00:00:00.000Z

Conditionally Required

locationCodestring

Request a Data Product for data from a specific Location

  • Location Code must be valid and match exactly, including case
  • Specific Location Codes can be obtained programatically from the locations service or interactively from the Available Locations page
locationCode=BACAX
deviceCategoryCodestring

Request a Data Product for data from devices belonging to a specific Device Category

deviceCategoryCode=ADCP2MHZ
deviceCodestring

Request a Data Product for data from a specific Device

  • Device Code must be valid and match exactly, including case
  • Specific Device Codes can be obtained programatically from the devices service or interactively from the Available Devices page
deviceCode=AandOpt0581
propertyCodestring

Request a Data Product for data for a specific Property

  • Property Code must be valid and match exactly, including case
  • Specific Property Codes can be obtained programatically from the properties service or interactively from the Available Properties page
property=pressure
Data Product Optionsstring

Each Data Product Extension may have required data product options that determine how to process or package the data.

  • Data Product Options include and are not limited to:
    • Quality Control
    • Resampling
    • Data Gaps
    • Hydrophone Data Diversion Mode
  • Data Product Options and usage information for each Data Product, can be found on the Data Product Options page

dpo_qualityControl=1

dpo_resample=average

dpo_average=600

Determining Search Type (Valid combinations)

Search Type Equivalent
Parameters Given
Instrument by location (Device Level)locationCode, deviceCategoryCode
Instrument by location (Sensor Level)locationCode, deviceCategoryCode, propertyCode
Instrument by Category (Device Level)deviceCode
Instrument by Category (Sensor Level)deviceCode, propertyCode
Variable by LocationlocationCode, propertyCode

 

Response

Detailed information will be returned if the files are in archive.  If the data product needs to be generated, only a dpRequestId will be returned. 

Success for files from Archive (HTTP 200)

 

{"compressedFileSize":12563408,"downloadTimes":{"10Mbps":7.076623,"50Mbps":1.4153247,"150Mbps":0.47177488},"dpRequestId":1852695,"fileSize":70766230,"numFiles":4}
Property
Type
Description
Example
dpRequestIdintegerReturns a numerical request id to be passed into the "run" method (SearchHdrId)
"dpRequestId":1852695
numFilesintegerReturns the number of files that meet the criteria of the search
"numFiles":4
fileSizeLongThe file size (in bytes)
"fileSize":70766230
compressedFileSizeLongThe file size (in bytes) after compression
"compressedFileSize":12563408
downloadTimesString':'FloatA set of three common download speeds and the time (in seconds) it would take to download the files for each speed
"downloadTimes":{"10Mbps":7.076623,"50Mbps":1.4153247,"150Mbps":0.47177488}

Success for generated data products (HTTP 200)

 

{"dpRequestId":1908432,"estimatedFileSize":"11940760","estimatedProcessingTime":"21"}
 
Property
Type
Description
Example
dpRequestIdintegerReturns a numerical request id to be passed into the "run" method (SearchHdrId)
"dpRequestId":1908432
estimatedFileSizestringThe estimated file size (in bytes)
"estimatedFileSize":"11940760"

or

"estimatedFileSize":"no estimated file size avalible."
estimatedProcessingTimestringThe estimated time it will take to process the data proeuct
"estimatedProcessingTime":"21"

or

"estimatedProcessingTime":"No processing time estimates found for this type of data."

Bad Request (HTTP 400)

errorCode
errorMessage
Description

25

Invalid Time Range, Start Time is in the future.

Occurs when the end is before the begin date/time.

  • The name of both of the datetime filters will be included in the "parameter" property
127Invalid parameter value

Occurs when an invalid code is used in the filter. Most filters require an exact match, otherwise this error will occur.

  • The name of the filter parameter will be included in the "parameter" property
128Missing parameter

Occurs when multiple parameters are needed, but not all are present. Occurs when begin is used without end

  • The names of the required filter parameters will be included in the "parameter" property separated by /
128

Requested type can not be determined

Occurs when the one of the 5 possible parameter combinations is not used.

  • A list of the valid parameter combinations are listed in the parameter

"possible parameter combinations are: locationCode, deviceCategoryCode; locationCode, deviceCategoryCode, propertyCode; locationCode, propertyCode; deviceCode; deviceCode, propertyCode"

129Invalid parameter name

Occurs when a filter parameter is used, but is not supported.



Examples

  • Request a 'Log File' data product in 'txt' format, using a Data Product Code of LF and Extension of 'txt' for Device Category Code of 'ADCP2MHZ' at the Location Code of 'BACAX', Between July 25th and 28th, 2016

http://data.oceannetworks.ca/api/dataProductDelivery?method=request&token=[YOUR_TOKEN_HERE]&locationCode=BACAX&deviceCategoryCode=ADCP2MHZ&dataProductCode=LF&extension=txt&begin=2016-07-25T00:00:00.000Z&end=2016-07-29T00:00:00.000Z

 

  • Request a 'Time Series Scalar Data' data product in 'csv' format, using a Data Product Code of 'TSSD' and Extension of 'csv' for the Property Code of 'salinity' at the 'Barkley Canyon Axis (Pod 1) location using the Location Code of 'BACAX', Between July 1st and 12th, 2017

http://data.oceannetworks.ca/api/dataProductDelivery?method=request&token=[YOUR_TOKEN_HERE]&locationCode=BACAX&propertyCode=salinity&dataProductCode=TSSD&extension=csv&begin=2017-01-01T00:00:00.000Z&end=2017-01-13T00:00:00.000Z&dpo_qualityControl=1&dpo_resample=none&dpo_dataGaps=0

run

The run method runs the data product created by a call to the request method.

Parameters

Parameter
Type
Description
Example
Required   
tokenstringAll Web Services require a token. This can be generated at http://dmas.uvic.ca/Profile. Click on the "Web Services" tab and click "Generate Token"token=5ba514e8-1203-428c-8835-879c8173e387
dpRequestIdinteger

A dpRequestId returned from the request method. (SearchHdrId)

dpRequestId=1852695

Response

Success (HTTP 200)

HTTP status code 200 is returned when all of the runs in the data product delivery request have completed running.

[
   {
      "dpRunId": 4521400,
      "status": "complete"
   }
]

 

The payload contains a list of runs, each run contains the following properties

Property
Type
Description
Example
dpRunId
integerA Run Id that can be used to download the data product when processing is complete. (AKA, SearchId)
"dpRunId":4521400
statusstringThe current status of data product request. Valid values are ("cancelled", "queued", "error", "running" and "complete")
"status": "complete"

Accepted (HTTP 202)

HTTP status code 202 is returned if all of the runs in the data product delivery request have not completed running.

[
   {
      "dpRunId": 4521404,
      "status": "Running"
   }
]

 

The payload contains a list of runs, each run can contain the following properties

Property
Type
Description
Example
dpRunId
integerA Run Id that can be used to download the data product when processing is complete. (AKA, SearchId)
"dpRunId":4521400
statusstringThe current status of data product request. Valid values are ("queued", "error" and "running")
"status": "complete"
queuePositionAn Integeronly returned if status is "queued". it starts at 1 meaning it is the first item in the queue
"queuePosition": 10
messagestringAn error message is returned when the status is "error" "message": "Unable to calculate position in queue."
 

Unauthorized (HTTP 401)

{
    "errors": [
        {
            "errorCode": 127,
            "errorMessage": "Invalid parameter value",
            "parameter": "token"
        }
    ]
}
errorCode
errorMessage
Description
127

Invalid parameter value

Occurs when an invalid token is used.

128Missing parameter

Occurs when the token parameter is not included in the request

 
 

Bad Request (HTTP 400)

{
    "errors": [
        {
            "errorCode": 129,
            "errorMessage": "Invalid parameter name",
            "parameter": "RequestId"
        },
        {
            "errorCode": 128,
            "errorMessage": "Missing parameter",
            "parameter": "dpRequestId"
        }
    ]
}
errorCode
errorMessage
Description
127

Invalid search header ID.

Occurs when an invalid Request ID is used in the dpRequestId paramters.

128Missing parameter

Occurs when multiple parameters are needed, but not all are present. Occurs when begin is used without end

  • The names of the required filter parameters will be included in the "parameter" property separated by /
129Invalid parameter name

Occurs when a filter parameter is used, but is not supported.

 

Example

http://data.oceannetworks.ca/api/dataProductDelivery?method=run&token=[YOUR_TOKEN_HERE]&dpRequestId=[YOUR_REQUEST_ID_HERE]

download

The download method downloads a file for the specified data product search. The file to download is specified by index with the first valid index being 1 and the last being the total number of files generated by the search.

Parameters

Parameter
Type
Description
Example
Required   
tokenstringAll Web Services require a token. This can be generated at http://dmas.uvic.ca/Profile. Click on the "Web Services" tab and click "Generate Token"token=5ba514e8-1203-428c-8835-879c8173e387
dpRunId integerThe dpRunId returned from the run method. (SearchId) 
indexintegerThe index of the file to be downloaded, valid values are 1 to the number of result files. If the index is greater than the number of result files a response code of 204, indicating no file at that index is returned. 

Response

Success

HTTP status code 200 will be returned in HTTP header and the requested file will be downloaded.

Response Codes

Response Code
Description
200 - OkThe file is included in the payload.
202 - AcceptedThe data product and index are valid but the data product is still being processed. This status can be used to poll waiting for data product completion. The payload includes the comment as displayed in the GUI.
204 - No ContentThere is no result file for the data product at the index specified. This can be used to stop looping through result files when the actual number of result files isn't known.
400 - Bad RequestMissing required parameters
 dpRunId is not a number
 index is not a positive number or not meta
 dpRunId is not a valid data product run Id
 The data product selected by dpRunId was run by a different user
 Index is meta but the data product does not include metadata
401 - UnauthorizedToken is invalid
404 - Not FoundIndex number is greater than the number of files
410 - GoneThe result file selected by dpRunId and index is not in expected location and may have been removed during FTP clean up.
500 - Internal Server ErrorThis is an unhandled exception and should not occur. All 500 errors should be immediately reported and a JIRA ticket created.

Examples

The following is an example of the return of data product that is still being processed, note that the response code in this case is 202.

http://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=[YOUR_TOKEN_HERE]&dpRunId=[YOUR_RUN_ID_HERE]&index=[YOUR_INDEX_#_HERE]

 

{
    "message": "Transferring (BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv) to the FTP server",
    "status": "running"
}

 

 

Property
Type
Description
Example
statusstringThe current status of data product request. Valid values are ("queued", "error" and "running")
"status": "running"
messagestringThe latest status message reported by the data product process. "message": "Transferring (BarkleyCanyon_Axis_ADCP2MHz_20160727T000005Z_20160731T235958Z-clean.csv) to the FTP server"

 

Code Examples

TitleCreatorModified
Python Client LibraryRyan Ross16-Aug-19
MATLAB Client LibraryRyan Ross26-Apr-19
Ouranos Use CaseAllan Rempel25-Apr-18
Bird Studies Canada Use CaseAllan Rempel24-Apr-18
Internal Use CaseRyan Ross24-Apr-18
Request Data ProductRyan Ross23-Apr-18

 

  • No labels