Aligned and Depth-Binned Profile Data (On-Demand)

Aligned and Depth Binned profile data that are collecting profile data are described here. This data product is generated on-demand / on-the-fly and with several improvements over Aligned and Depth-Binned Profile Data (Legacy). A profile is a series of measurements made throughout the water column grouped by each pass through the water, known as a 'cast', as opposed to continuous time series data. Profiles of the water column can be created by performing casts with instruments. Casts involve lowering an instrument down into the water and back up again while it is making measurements. Cast detection or delineation is performed automatically with manual oversight, contact us for further details. The measurements from the various sensors attached to and within the instrument are processed to filter, smooth, align, and average data into depth bins. Information about the data processing is included in the header of the data product. Specifically, the default processing steps are:

The alignment of oxygen to account for instrument response lags. This is instrument dependent.
Chlorophyll is filtered using a 25 median point filter.
Temperature and conductivity data is smoothed using a 5 point running mean.
Practical salinity, sound speed, and density are calculated using smoothed temperature, smoothed conductivity, and pressure.
Data is averaged into 1 m bins centred at 1 m intervals. Data is only averaged into its corresponding bin if 70% of the data within the bounds of the bin has a QAQC flag of 1.

Generally, the descending portion of the cast is then extracted (this will be user configuration in the future, as well as cast detection, alignment, binning parameters). This data product is only used with aligned and depth binned data. To distinguish it from the Aligned and Depth-Binned Profile Data (Legacy) data product, these files are named without "-CORRECTED" added to the standard file-name. This data product includes data from connected sensors when applicable. If connected sensors have been included. this is denoted by having "-ALLCONNECTEDSENSORS" add to the standard file-name. This data product retrieves all aligned and depth binned casts within the search range and produces one output file with one or more casts within it. Each cast is prefaced by a header which contains additional information associated with the cast. An example of this header can be seen below. The columns of data are labelled with numbers counting from left to right. Each number is labelled with the type of data contained within it and the associated units. The columns of data are comma-delimited with one space on either side of the comma. This data product is very similar to Aligned and Depth-Binned Profile Data (Legacy), however, each cast and instrument in the search range is included in one file (up to file and memory size limits).

Please note that these files are created on-demand from aligned and depth-binned sensors (sensor and device-level) from devices operating as profilers. Currently, availability is limited to pre-processed cast delineations, which will be back-filled for all data.

Please consult individual instruments' documentation for more information.

Oceans 3.0 API filter: dataProductCode=OTFCPD

Example Cast Header

Deployment 1 of 1
Patrol name: Salish Sea Marine Survival Program Patrol 3
NumberOfCasts: 5 casts in epoch in search range requested
Cast: 1 of 5
ProcessingCom: Calculation of derived variables: Practical Salinity, Density and Sound Speed are calculated using smoothed Conductivity and smoothed scan-shifted Temperature. Temp Scan shift ahead = 0; Smoothing: 5-point running mean.; 25 median point filter applied to the Chlorophyll data.; Oxygen data shifted ahead by 54 scans.; All data binned into depth bins of: 1m.; Percent Good Acceptance Value: 70.;
Cast location: Community Fishers
Station name: IRS3
ProcessingDate: 2019-10-29 22:17:32.606 ; UTC
StartDateCast: 2018-05-10 15:54:04 ; UTC
EndDateCast: 2018-05-10 16:00:44 ; UTC
LongitudeCastStart: -124.1091867 ; deg E
LongitudeCastEnd: -124.1097939 ; deg E
LatitudeCastStart: 49.546836 ; deg N
LatitudeCastEnd: 49.5468715 ; deg N
DepthCastStart: 1 m
DepthCastEnd: 151.5 m
*** 4 devices in cast
RBRconcerto C.T.D.DO.Fllfast6 (S/N 65647)
Aanderaa Optode 4831F (S/N 300)
Turner Cyclops-7 Fluorometer (S/N 2103465)
Samsung Galaxy Tab S2 (SN R52HA2KMCMP)
*** end devices
Column 1: Time, Column 2: Absolute Pressure (aligned and depth-binned); (decibar), Column 3: Conductivity (aligned and depth-binned); (S/m), Column 4: Density (aligned and depth-binned); (kg/m3), Column 5: Depth (aligned and depth-binned) (mobile position/attitude sensor); (m), Column 6: Practical Salinity (aligned and depth-binned); (psu), Column 7: Pressure (aligned and depth-binned); (decibar), Column 8: Sound Speed (aligned and depth-binned); (m/s), Column 9: Temperature (aligned and depth-binned); (C), Column 10: Oxygen Saturation (aligned and depth-binned); (%), Column 11: Chlorophyll (aligned and depth-binned); (ug/l), Column 12: Latitude (aligned and depth-binned) (mobile position/attitude sensor); (deg), Column 13: Longitude (aligned and depth-binned) (mobile position/attitude sensor); (deg)
------ BEGIN DATA ------
20180510T155405.333Z , 9.609099 , 2.787437 , 1016.257904 , 1.000000 , 22.154511 , 1.229099 , 1489.057186 , 14.175684 , 106.587980 , 3.951083 , 49.546836 , -124.109188

Behaviour when no data is retrieved

If the requested search range does not contain any data a file is created with "Missing Data Detected" for each sensor column in the file. These "Missing Data Detected" files are generated for each deployment in the search range. An example of this type of missing data file can be seen below.

Example Cast When No Data Retrieved

Deployment 1 of 1
Patrol name:   CommunityFishers_Set003_2020-01
NumberOfCasts: No casts in epoch in search range requested
Cast: 0 of 0
ProcessingCom: Calculation of derived variables: Practical Salinity, Density and Sound Speed are calculated using smoothed Conductivity and smoothed scan-shifted Temperature. Temp Scan shift ahead = 0; Smoothing: 5-point running mean.; 25 median point filter applied to the Chlorophyll data.; Oxygen data shifted ahead by 54 scans.; All data binned into depth bins of: 1m.; Percent Good Acceptance Value: 70.;
Cast location: Community Fishers
Station name:  
ProcessingDate: Not Available ; UTC
StartDateCast: 2020-02-11 00:00:00 ; UTC
EndDateCast:   2020-02-11 09:00:00 ; UTC
LongitudeCastStart: Not Available ; deg E
LongitudeCastEnd: Not Available ; deg E
LatitudeCastStart:Not Available ; deg N
LatitudeCastEnd: Not Available ; deg N
DepthCastStart: Not Available
DepthCastEnd: Not Available
*** 4 devices in cast
AML CTD Plus X 50222
Turner Cyclops-7 Fluorometer (S/N 900052)
Aanderaa Optode 4531F-IW (S/N 737)
XSlate D10 Tablet (S/N SY9560FS00338)
*** end devices
Column 1: Time, Column 2: Conductivity (aligned and depth-binned); (S/m), Column 3: Density (aligned and depth-binned); (kg/m3), Column 4: Depth (aligned and depth-binned); (m), Column 5: Practical Salinity (aligned and depth-binned); (psu), Column 6: Turbidity (aligned and depth-binned); (NTU), Column 7: Pressure (aligned and depth-binned); (decibar), Column 8: Sound Speed (aligned and depth-binned); (m/s), Column 9: Temperature (aligned and depth-binned); (C), Column 10: Oxygen Saturation (aligned and depth-binned); (%), Column 11: Chlorophyll (aligned and depth-binned); (ug/l)
------ BEGIN DATA ------
Missing Data Detected, Missing Data Detected, Missing Data Detected, Missing Data Detected, Missing Data Detected, Missing Data Detected, Missing Data Detected, Missing Data Detected, Missing Data Detected, Missing Data Detected, Missing Data Detected
------ END DATA ------

Revision History

20191107: Initial Release

Formats

This data product is available in two formats: cor and csv. COR is a text document with a .cor extension and can be opened in a wide variety of programs such as notepad, Wordpad, etc. CSV is a text document with comma-separated values and a .csv extension, it can be opened in various programs such as excel, google sheets, etc.

Oceans 3.0 API filter: extension=cor

OR

Oceans 3.0 API filter: extension=csv

Data Product Options

There are two data product option sets for this data product. These options are shown to represent the processing that is always applied to this data product and is therefore always selected. Also, there are options to configure the processing parameters used when calculating Aligned and Depth Binned sensors.

For Aligned and Depth-Binned Profile Data (On-Demand) data products on select Conductivity Temperature Devices only (vertically profiling devices deployed on mechanical profilers and with the Community Fishers project.

Devices with connected sensors, also known as 'piggyback' sensors, will have two options as shown below, while devices without connected sensors will have one selected data product option. These are fixed options, included for compatibility within the code and for visibility to users in Data Search (unfortunately, they are still required parameters for the dataProductDelivery API).

Oceans 3.0 API filter: dpo_includePiggybackSensors=0&dpo_sensorstoinclude=1

For Cast Scalar Multi-Profile Plot and Aligned and Depth-Binned Profile Data (On-Demand) data products on Conductivity Temperature Depth devices

Selecting non-default values for either the Cast Processing Bin Size or Cast Processing Percent Acceptance for Bin will result in Aligned and Depth Binned data being calculated on the fly from cleaned raw data and will take longer to generate a data product as a result. Files that contain data calculated on the fly will have "-onTheFly" in their filename, and the ProcessingDate header information will be the time of file generation in UTC. For files calculated on the fly with no binning selected then the column labels will contain "aligned" after the sensor name. For files calculated on the fly with binning selected the column labels will contain "aligned and depth-binned" after the sensor name.

Cast Processing Bin Size

This data product option controls the size of the bins the data is averaged into in meters. The default bin size is one-meter bins. Data is only averaged into its corresponding bin if the number of data points within the bounds of a bin with a QAQC flag of 1 meets the percent acceptance threshold (the default percent acceptance value is 70%). The bin size can be user set between zero meters to ten meters. The "No Binning" option can also be selected which will result in the data not being averaged.

For example, a bin size of 2 meters will have bin bounds from 1m to 3m, 3m to 5m, 5m to 7m, etc. The data within these depth bounds are averaged into one value if the percent acceptance threshold is met, which places the averaged values at 2m, 4m, 6m, etc.

Default - Oceans 3.0 API filter: dpo_binSize=-1
No Binning - Oceans 3.0 API filter: dpo_binSize=-2
Configurable Bin Size - Oceans 3.0 API filter: dpo_binSize=0 to dpo_binSize=10

Cast Processing Percent Acceptance for Bin

This data product option controls the percent acceptance threshold applied to data during the binning process. The default percent acceptance value is 70%. Data is only averaged into its corresponding bin if the number of data points within the bounds of a bin with a QAQC flag of 1 meets the percent acceptance threshold which is user-configurable with this data product option. The percent acceptance value can be set between zero percent to one hundred percent.

For example, if 7 data points out of 10 in a bin have a QAQC flag of 1 and the percent acceptance threshold is set to 70% the averaged value for the bin will be calculated from the 7 data points with a QAQC flag of 1. If the percent acceptance value is 80% then no value will be calculated for the bin as less than 80% of the data within the bin is has a QAQC flag of 1.

Default - Oceans 3.0 API filter: dpo_percentAcceptance=-1
Configurable Percent Acceptance - Oceans 3.0 API filter: dpo_percentAcceptance=0 to dpo_percentAcceptance=100

For Aligned and Depth-Binned Profile Data (On-Demand) data products on Conductivity Temperature Depth and piggyback devices

This option controls whether to produce separate .cor files for each cast in a search range or to continue writing into one file until memory limits are reached. The default behavior is to write cast data into one file until memory limits are reached.

None (break on memory limits only)
Oceans 3.0 API filter: dpo_fileBreaks=0

Produce One File Per Cast
Oceans 3.0 API filter: dpo_fileBreaks=1

For Aligned and Depth-Binned Profile Data (On-Demand) data products on select Conductivity Temperature Devices only.

Oceans 3.0 API filter: sensorstoinclude=1

Example Files

An example file for a device with connected sensors is available to download here: RBRCONCERTO65647_20180510T155405Z_20180510T215123Z-ALLCONNECTEDSENSORS.cor

An example file for a device without connected sensors is available to download here: AandOpt4831F300_20180510T155405Z_20180510T215123Z.cor

Page tree

152