View Source

These methods allow users to directly download previously generated data product files in our files archive. ONC scripts auto-generate and archive files of different types for device data at set time intervals; These archived files can be downloaded without waiting for a generation process to finish (i.e. Data product download methods).

Archived files have a unique filename (i.e. "NAXYS_HYD_007_20091231T235919.476Z-spect.png") that includes the device code ("NAXYS_HYD_007") and the UTC date & time when the data in the file started being measured ("20091231T235919.476Z")

Due to military security regulations, some very recent files (i.e. hydrophone .wav files in the last hour) might not be made immediately available

Summary

Method	Description
`getListByLocation (filters, allPages)`	Get a list of archived files for a device category in a location	link
`getListByDevice (filters, allPages)`	Get a list of archived files for a device	link
`getFile (filename, overwrite)`	Download a file with the given filename	link
`getDirectFiles (filters, overwrite, allPages)`	Download a list of archived files that match the filters provided	link

getListByLocation

Obtains a list of archived files for a device category in a location. The filenames obtained can be used to download files using the getFile() method.

Parameter

Type

Description

Example

filters

dictionary

Dictionary of filters used to specify the data origin, to be provided to the getListByLocation method from the API archivefiles service.

The following filters are mandatory:

The following filters are optional:

dateFrom
dateTo
dateArchivedFrom
dateArchivedTo
returnOptions
rowLimit
page
getLatest
extension

The "extension" filter allows you to limit the results to filenames that have the given file extension (i.e. "mp3", "wav", "mat", etc.). This filter is provided by the client library (not in the API).

Visit the getListByLocation method for more information on filter usage.

{ 'locationCode':'BACAX',
'deviceCategoryCode':'ADCP2MHZ',
'dateFrom':'2016-07-27T00:00:00.000Z',
'dateTo':'2016-08-01T00:00:00.000Z',
}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

False: Only download the first page of data
True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'locationCode':'BACAX',
    'deviceCategoryCode':'ADCP2MHZ',
    'dateFrom':'2016-07-27T00:00:00.000Z',
    'dateTo':'2016-08-01T00:00:00.000Z',
}

result = onc.getListByLocation(filters, allPages=True)
onc.print(result)

Returns

A dictionary with a list of the filenames of the archived files found for the filters provided.

{
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/archivefiles?method=getListByLocation&(...)",
    "files": [
        "BC_POD1_AD2M_20160727T000000.000Z.txt",
        "BC_POD1_AD2M_20160728T000000.000Z.txt",
        "BC_POD1_AD2M_20160729T000000.000Z.txt",
        "BC_POD1_AD2M_20160730T000000.000Z.txt"
    ]
}

By default, only a string filename is returned for each file. Setting the "returnOptions" filter allows you to increase the information returned for each file in the "files" list, as shown in the examples below (warning: this might noticeably increase the request size for long lists):

"files": [
    {
        "archiveLocation": "/ONC_Archive",
        "dateFrom": "2016-07-27T00:00:00.000Z",
        "filename": "MYDEV_20160727T000000.000Z.txt",
        "path": "15/32/19"
    },
    (...)
]

"files": [
    {
        "archivedDate": "2016-08-24T21:51:42.000Z",
        "compression": "gz",
        "dataProductCode": "LF",
        "dateFrom": "2016-07-27T00:00:00.000Z",
        "dateTo": "2016-07-28T00:00:00.000Z",
        "deviceCode": "BC_POD1_AD2M",
        "fileSize": 6269663,
        "filename": "MYDEV_20160727T000000.000Z.txt",
        "modifyDate": "2016-08-24T21:51:48.000Z",
        "uncompressedFileSize": 33359040
    },
    (...)
]

If there are additional data pages to download, the "next" property will contain a dictionary with information to obtain the next page of data as shown in the following example:

{
    "queryUrl": "https://data.oceannetworks.ca/api/archivefiles?locationCode=TWDP&(...)",
    "files": [ (...) ],
    "next": {
            "method": "getListByLocation",
            "token": "YOUR_TOKEN",
            "locationCode": "TWDP",
            "deviceCategoryCode": "TSG",
            "dateFrom": "2016-09-01T00:00:37.040Z",
            "dateTo": "2016-09-01T00:01:00.000Z",
            "page": "2"
    }
}

The method returns a dictionary with the keys described below.

Key Type Description

queryUrl string The URL that obtained the results

files list A list of archived files found. By default a list of strings with filenames; when "returnOptions" is specified, this key will hold a list of dictionaries instead.

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data.

If this is the last page of data, "next" will be null.

If "returnOptions" is set to "archiveLocation", each file found is a dictionary with the following keys:

Key	Type	Description	Example
filename	string	The name of the file	`"BC_POD2_JB_20090908T000000.000Z.txt"`
dateFrom	string	The beginning date & time for the data included in the file	`"2010-04-21T00:00:00.000Z"`
archiveLocation	string	Root location of the archived file	"/ONC_Archive"
path	string	Path for this file in the ONC archive	`"14/38/22"`

If "returnOptions" is set to "all", each file found is a dictionary with the following keys:

Key	Type	Description	Example
filename	string	The name of the file	`"BC_POD2_JB_20090908T000000.000Z.txt"`
compression	string	The file extension if the file is compressed in the archive	`"gz"`
dataProductCode	string	The data product code that generates this file	`"LF"`
deviceCode	string	Device code for the origin of this file	`"BC_P0D1_AD2M"`
dateFrom	string	The beginning of the data time range included in this file	`"2010-04-21T00:00:00.000Z"`
dateTo	string	The end of the data time range included in this file	`"2010-04-22T00:00:00.000Z"`
archivedDate	string	The UTC timestamp when the file was archived	`"2010-04-21T00:00:00.000Z"`
modifyDate	string	The last timestamp of any modification related to the file	`"2010-04-22T01:55:00.000Z"`
fileSize	int	File size (bytes) in the archive after compression	`6349499`
uncompressedSize	int	Original file size (bytes) before compression	`156545`

getListByDevice

Obtains a list of archived files from a device. The filenames obtained can be used to download files using the getFile() method.

Parameter

Type

Description

Example

filters

dictionary

Dictionary of filters used to configure the data origin, to be provided to the getListByDevice method from the API archivefiles service.

The filter deviceCode is mandatory and sets the data source.

The following filters are optional:

dateFrom
dateTo
dateArchivedFrom
dateArchivedTo
returnOptions
rowLimit
page
getLatest
extension

The "extension" filter allows you to limit the results to filenames that have the given file extension (i.e. "mp3", "wav", "mat", etc.). This filter is provided by the client library (not in the API).

Visit the getListByDevice method for more information on filter usage.

{'deviceCode':'NAXYS_HYD_007',
'dateFrom': '2010-01-01T00:00:00.000Z',
'dateTo' : '2010-01-01T00:00:10.000Z'}

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

False: Only download the first page of data
True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'deviceCode':'NAXYS_HYD_007',
    'dateFrom': '2010-01-01T00:00:00.000Z',
	'dateTo'  : '2010-01-01T00:00:10.000Z'
}
result = onc.getListByDevice(filters, allPages=True)
onc.print(result)

Returns

A dictionary with a list of the filenames of the archived files found for the filters provided.

{
    "next": null,
    "queryUrl": "https://data.oceannetworks.ca/api/archivefiles?method=getListByDevice&(...)",
    "files": [
        "BC_POD1_AD2M_20160727T000000.000Z.txt",
        "BC_POD1_AD2M_20160728T000000.000Z.txt",
        "BC_POD1_AD2M_20160729T000000.000Z.txt",
        "BC_POD1_AD2M_20160730T000000.000Z.txt"
    ]
}

By default, only a string filename is returned for each file. Setting the "returnOptions" filter allows you to increase the information returned for each file in the "files" list, in the same way as explained for the Archive file download methods method.

The method returns a dictionary with the keys described below.

Key Type Description

queryUrl string The URL that obtained the results

files list A list of archived files found. By default a list of strings with filenames; when "returnOptions" is specified, this key will hold a list of dictionaries instead.

If the data requested exceeds the row limit, this dictionary contains the parameters required to produce the next page of data.

If this is the last page of data, "next" will be null.

If "returnOptions" is set to "archiveLocation", each file found is a dictionary with the following keys:

Key	Type	Description	Example
filename	string	The name of the file	`"BC_POD2_JB_20090908T000000.000Z.txt"`
dateFrom	string	The beginning date & time for the data included in the file	`"2010-04-21T00:00:00.000Z"`
archiveLocation	string	Root location of the archived file	"/ONC_Archive"
path	string	Path for this file in the ONC archive	`"14/38/22"`

If "returnOptions" is set to "all", each file found is a dictionary with the following keys:

Key	Type	Description	Example
filename	string	The name of the file	`"BC_POD2_JB_20090908T000000.000Z.txt"`
compression	string	The file extension if the file is compressed in the archive	`"gz"`
dataProductCode	string	The data product code that generates this file	`"LF"`
deviceCode	string	Device code for the origin of this file	`"BC_P0D1_AD2M"`
dateFrom	string	The beginning of the data time range included in this file	`"2010-04-21T00:00:00.000Z"`
dateTo	string	The end of the data time range included in this file	`"2010-04-22T00:00:00.000Z"`
archivedDate	string	The UTC timestamp when the file was archived	`"2010-04-21T00:00:00.000Z"`
modifyDate	string	The last timestamp of any modification related to the file	`"2010-04-22T01:55:00.000Z"`
fileSize	int	File size (bytes) in the archive after compression	`6349499`
uncompressedSize	int	Original file size (bytes) before compression	`156545`

getFile

Downloads a file from the archive, identified by its unique filename.

Parameter

Type

Description

Example

filename

string

The filename for the file requested

"NAXYS_HYD_007_20091231T235919.476Z.wav"

overwrite

boolean

Whether new files downloaded will overwrite previous files with the same filename found in the output directory.

True: Overwrite files with the same filename
False: Do not overwrite files (files with the same filename will be skipped)

Default: False

True

(parameters with an underline are required)

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

result = onc.getFile('NAXYS_HYD_007_20091231T235919.476Z.wav')
onc.print(result)

Returns

A dictionary with information on the file download result.

{
    "url": "https://data.oceannetworks.ca/api/archivefiles?filename=NAXYS_HYD_007_(...)",
    "status": "completed",
    "size": 10240044,
    "downloadTime": 1.535,
    "file": "NAXYS_HYD_007_20091231T235919.476Z.wav"
}

The response is a dictionary with the following keys:

Key	Type	Description
url	string	The URL used to download the file
status	string	The status of the download process for this file. Possible values are: `complete`: The file was downloaded `error`: The file was not downloaded due to an error `skipped`: The file was not downloaded because a file with the same name already exists in the output path and "`overwrite`" is set to `False`
file	string	The filename of the file if it was downloaded (otherwise, an empty string)
size	int	The size of the downloaded file in bytes (0 if it was not downloaded)
downloadTime	float	The time duration (in seconds) spend downloading this file (0 if it was not downloaded)

getDirectFiles

Downloads all archived files that match the filters specified. Will also print messages to the console describing the download progress.

Internally, this function invokes the method getListByLocation() or getListByDevice() to obtain a list of files, and then the method getFile() to download all the files found.

Parameter

Type

Description

Example

filters

dictionary

Dictionary of filters used to configure the request.

It is mandatory to provide either of the following filters to identify the data origin:

locationCode and deviceCategoryCode to download files from a device category in a location using getListByLocation()
deviceCode to download files from a device using getListByDevice()

The following filters are optional:

dateFrom
dateTo
dateArchivedFrom
dateArchivedTo
returnOptions
rowLimit
page
getLatest
extension

The "extension" filter allows you to limit the results to filenames that have the given file extension (i.e. "mp3", "wav", "mat", etc.). This filter is provided by the client library (not in the API).

Visit getListByLocation and getListByDevice for more information on filter usage.

{'deviceCode':'NAXYS_HYD_007',
'dateFrom': '2010-01-01T00:00:00.000Z',
'dateTo' : '2010-01-01T00:00:10.000Z'}

overwrite

boolean

Whether new files downloaded will overwrite previous files with the same filename found in the output directory.

True: Overwrite files with the same filename
False: Do not overwrite files (files with the same filename will be skipped)

Default: False

True

allPages

boolean

Whether this method should automatically download all pages when the data requested exceeds the row limit and requires to be downloaded in multiple pages.

False: Only download the first page of data
True: If required, keep downloading all pages of data until all the requested data is obtained.

Default value: False

True

(parameters with an underline are required)

from onc.onc import ONC
onc = ONC('YOUR_TOKEN_HERE')

filters = {
    'deviceCode':'NAXYS_HYD_007',
    'dateFrom': '2010-01-01T00:00:00.000Z',
	'dateTo'  : '2010-01-01T00:00:10.000Z'
}
result = onc.getDirectFiles(filters, allPages=True)
onc.print(result)

Returns

A dictionary with information on the file download results and metrics on the download process, as in the following example:

{
    "downloadResults": [
        {
            "url": "https://data.oceannetworks.ca/api/archivefiles?filename=(...)",
            "status": "completed",
            "size": 10240044,
            "downloadTime": 1.223,
            "file": "NAXYS_HYD_007_20091231T235919.476Z.wav"
        },
        (...)
    ],
    "stats": {
        "totalSize": 10821184,
        "downloadTime": 1.743,
        "fileCount": 6
    }
}

Any downloaded files with "status": "completed" should also appear in the "outPath" directory.

The response is a dictionary with the following keys:

Key	Type	Description
downloadResults	list	A list with dictionaries, with the download results for each file (see below)
stats	dictionary	A dictionary with metrics on the overall download process
stats.totalSize	string	The total size of all files downloaded (in bytes)
stats.downloadTime	int	The time duration (in seconds) spend downloading all files
stats.fileCount	float	The number of files downloaded

The "downloadResults" list contains dictionaries with the download results for each file, with the following keys:

Key	Type	Description
url	string	The URL used to download the file
status	string	The status of the download process for this file. Possible values are: `complete`: The file was downloaded `error`: The file was not downloaded due to an error `skipped`: The file was not downloaded because a file with the same name already exists in the output path and "`overwrite`" is set to `False`
file	string	The filename of the file if it was downloaded (otherwise, an empty string)
size	int	The size of the downloaded file in bytes (0 if it was not downloaded)
downloadTime	float	The time duration (in seconds) spend downloading this file (0 if it was not downloaded)