This documentation details how to download data product files that resulted from a search using the User Directory, FTP (File Transfer Protocol) and API (Application Programming Interface). 

Table of Contents:

Accessing files via User Directory

To view a user search folder make sure you are logged into Oceans 3.0 then, in your web browser:

  1. Click "More"
  2. Click "User Directory"


You will be presented with a folder structure containing search folders similar to this, with a URL of the form: "ftp.oceannetworks.ca/pub/user*****/" where ***** is your userId.

If you have many searches in your User Directory, it can be difficult to find the one you want. You can sort the page by Last Modified, or you can look up the folder name from Data Search's completed cart. For the example above, the user would see a download link "Search_38454349.zip" under the Download column in Data Search's completed carts, so the folder name would be "search38454349". In the future, we may add a direct link from the Data Search cart to the search folder.

If you only want to download one file from a search folder then you do not need to use an FTP client. Navigate into the search folder and click the filename of the file you wish to download, which will initiate the download via your web browser.

Downloading a search folder or files using an FTP client

If you have a search folder with too many files to individually download then the whole folder or a range of files can be downloaded using an FTP client. FTP client software is specifically designed to download large amounts of data and files. and tends to be more reliable than using your web browser.

FTP client

First, an FTP client is required, this how-to will use FileZilla which is free, open source, and available from: https://filezilla-project.org/

Procedure

Once you have an FTP client installed and have located the search folder that you would like to download:

  1. Within the "Host" field of the FTP client put: ftp.oceannetworks.ca
  2. Leave "Username", "Password", and "Port" empty.
  3. Select "Quickconnect".
  4. Within the "Local site" pane navigate to a target folder for the downloaded files within your local file system.
  5. Copy and paste the "/pub/user*****/" portion of the FTPs URL into the "Remote Site" field. 
  6. Within the "Remote Site" pane right click on the folder you want to download and select download. Alternatively, open the folder and select the range of files to download.
  7. Check on the status of the download in the lower pane.
Example screen grab from FileZilla

Notes on the contents of the User Directory

When viewing or downloading the entire search folder, you will see additional files that are not present in the usual .zip file download from the Data Search cart. The extra files will usually include the ISO 19115 xml metadata file and a text file that captures all the search status information, called searchStatusUpdatesLog.txt. There may also be some leftover ancillary files that weren't cleaned up during data product generation, although this is very rare. 

Important: when searches retrieve files directly from the archive, they will not appear in the search folder. These files are only available via the the Data Search .zip file download or via the API dataProductDelivery download method. Some searches will generate some files on-the-fly in the search folder while also having some files retrieved from the archive. The search does this to fill in gaps where pre-generated files haven't been archived yet. The searchStatusUpdatesLog.txt file will have some info on this, for search types where it is generated.

Downloading search result files using the API

The API dataProductDelivery download method can be used simply in your web browser to download data product files one at a time, however, users can also make use of the ONC Client Libraries or write their own code to wrap the following procedure in a loop to get all the files. The API dataProductDelivery download method takes your token (available in the Web Services API tab here) and the searchID as inputs, users can call it, incrementing the file index until they get a no file found / error code 204 back. The searchID, or dpRunId as it's known in the API, is the numeric part of the search folder name as demonstrated above. Users can do this just in their browser, using this example URL: https://data.oceannetworks.ca/api/dataProductDelivery?method=download&token=[YOUR_TOKEN_HERE]&dpRunId=[YOUR_RUN_ID_HERE]&index=[YOUR_INDEX_#_HERE] This would be extremely tedious and error prone, which is why it's much easier to wrap this in a loop in your favourite programming language, such as Python, MATLAB, R, etc, to increment the file index number. This is also exactly what theONC Client Libraries do. The libraries have a download function that handles this, just supply your token and searchID/dpRunId, see the documentation here (this documentation is being phased out in favour of documentation hosted in GitHub here).

More help needed?

If you have questions, use the request support form in Oceans 3.0 or Contact Support.


  • No labels