BioSonics Time Series
Data products for BioSonics Echosounders are described here. Please note, new data products become available every day shortly after the log file is closed and processed at midnight UTC (4 or 5 PM Pacific time). The conversion from log to DT4 formats takes about 20 minutes for one day of data, as does the conversion from DT4 to MAT. Once the data is converted, the DT4 and MAT files are archived for quick retrieval, (not all MAT files are currently archived, so conversion is done on demand).
Oceans 2.0 API filter:
- 20110618: Initial data products released in manufacturer DT4 and MATLab mat formats. This initial version is based on Rick Pawlowicz's code, with additions from Asa Parker (BioSonics) and others
- 20140420: Improved speed
This data is available in DT4 and MAT formats. Content descriptions and example files are provided below.
To produce the file, the following requirements apply:
- A new DT4 file is started at the beginning of each day (midnight UTC exactly), or when the driver is restarted (this should account for configuration changes, site changes, etc).
- The instrument date/time field is replaced using the ONC timestamp at the beginning of the log file (since this timestamp is more accurate than the instrument clock).
- MAT files are made from DT4 files. MAT files are made on the hour every hour, except when the data is truncated, then small files are created. The date-from and date-to of a DT4 file and the MAT files created from it may not agree if the DT4 file doesn't contain data for it's full time range.
- Although each channel/frequency pings simultaneously, simultaneous ping packets will have slightly different time stamps, usually separated by 100 milliseconds. This means simultaneous ping packets may end up in different DT4 files if they occur on either side of midnight. The only way to sort this out is to look at the times in the MAT files.
This format is specific to the manufacturer, and consists of sections called tuples (e.g., channel descriptor, time and ping tuples). When using BioSonics data acquisition software, data is normally stored in this way. Although we use custom-built drivers to communicate with our instruments, we can use the raw data in the log file to produce the DT4 file which can be interpreted by BioSonics post-processing software (most recent version available from BioSonics website) in play-back mode, and commercial software like EchoView.
A new file is generated each day, and whenever the driver is restarted. The reference time in the time tuple and the ping time in the ping tuples correspond to the times the measurements were received at the ONC Canada shore station (in place of the internal instrument clock values).
The format and tuple contents are further described in the manufacturer's documentation. However, there are two tuples that are not described in this document, but are regularly included in DT4s.
pulse tuple (0X0013) - describes the pulse type of the data for each configuration
marker tuple (0X0010 and 0X0020) - used to indicate the start and end of the time & ping tuples
Oceans 2.0 API filter: extension=dt4
MAT files (v7.3) can be opened using MathWorks MATLAB 7.3 (R2006b) or later. These files use 64-bit indexing, but will work on a 32-bit operating system (will be slower). A new file a created every hour from the start of the parent DT4 file(s). One hour files were selected so that the memory requirements are not too onerous. Opening a one-hour BioSonics MAT file in MATLAB will require about 500 MB of contiguous available memory. Various plotting routines and analysis may cause MATLAB to use up to 2 GB of memory.
Each MAT file contains the following structures: meta, data, units. Entries in italic not applicable to the type of BioSonics Echosounders deployed, but are standard data fields in the DT4 file specification.
data: structured array containing the BioSonics data (one structure per configuration in cycle), having the following fields which pertain to the dt4 contents (for details, refer to the manufacturer manual).
name: serial number
frequency: transducer frequency (Hz)
time: datenum vector, UTC timestamp (time source is ONC shore station)
range: vector of distance to each bin
pingnum: ping number
vals: amplitude of return signal (A/D Counts - main data matrix - data type uint32)
nsamps: number of samples in the ping tuples (read from the ping tuple)
env: structure containing environmental configuration values
- absorb: configuration absorption coefficient (dB/m - specific to each channel/frequency)
- sv: configuration sound velocity (m/s)
- temperature: configuration temperature (degrees C)
- salinity: configuration salinity (ppt)
- power: transmit power reduction factor
- numChan: total number of frequencies/channels/configurations in the dt4 file
snd: structure containing details of the channel descriptor tuple
- address: configuration/channel number
- npings: total number of pings in the dt4
- npingsExtracted: total number of pings extracted from the dt4
- sampperping: samples per ping (this is usually equal to max(data.nsamps) )
- sampperiod: time between successive samples in a ping (microseconds)
- pingperiod: number of seconds between successive pings
- pulselen: duration of transmitted pulse (milliseconds)
- blank: initial blanking (number of samples skipped before data recording begins for each ping)
- threshold: data collection threshold (dB)
- rxee: receiver EEPROM image structure (can ignore split-beam options since that feature is not included in this instrument)
- ssn: transducer SN
- calDateNum: last calibration date (Matlab datenum)
- calDateStr: last calibration date (datestring)
- calTech: initials of technician performing calibration
- sl: source level (0.1 dB)
- rs: receiver sensitivity for main element (0.1 dB)
- rsw: receiver sensitivity for wide-beam (0.1 dB)
- pdpy: sign of split beam y-axis (zero=positive, nonzero=negative)
- pdpx: sign of split beam x-axis (zero=positive, nonzero=negative)
- noiseFloor: noise floor of transducer (AD Counts) due to system noise
- transducerType: transducer hardware type (0 for single-beam, 3 for dual-beam, 4 for split-beam)
- frequency: transducer frequency (Hz)
- pdy: absolute value of the split-beam y-axis element separation (0.01 mm)
- pdx: absolute value of the split-beam x-axis element separation (0.01 mm)
- phpy: split-beam y-axis element polarity
- phpx: split-beam x-axis element polarity
- aoy: split-beam y-axis angle offset (0.01 degrees)
- aox: split-beam x-axis angle offset (0.01 degrees)
- bwy: y-axis -3dB one-way beam width for the main element (0.1 degrees)
- bwx: x-axis -3dB one-way beam width for the main element (0.1 degrees)
- sampleRate: sampling rate of the A/D converter (Hz), usually 41667.
- bwwwy: y-axis -3dB one-way beam width for wide-beam or phase elements (0.1 degrees)
- bwwx: x-axis -3dB one-way beam width for wide-beam or phase elements (0.1 degrees)
- phy: split-beam y-axis phase aperture (0.1 degrees)
- phx: split-beam x-axis phase aperture (0.1 degrees)
- ccor: calibration correction
opmode: operating mode
units: structure containing unit of measure for fields in structures above. For instance, units.lat='degrees N'.
- deviceID: A unique identifier to represent the instrument within the Ocean Networks Canada data management and archiving system.
- creationDate:Date and time (using ISO8601 format) that the data product was produced. This is a valuable indicator for comparing to other revisions of the same data product.
- deviceName: A name given to the instrument.
- deviceCode: A unique string for the instrument which is used to generate data product filenames.
- deviceCategory: Device category to list under data search ('Echosounder').
- deviceCategoryCode: Code representing the device category. Used for accessing webservices, as described here: API / webservice documentation (log in to see this link).
- lat: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site latitude and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure..
- lon: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site longitude and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
- depth: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site depth and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
- deviceHeading: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site heading and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
- devicePitch: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site pitch and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
- deviceRoll: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site roll and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
- siteName: Name corresponding to its latitude, longitude, depth position.
- stationCode: Code representing the station or site. Used for accessing webservices, as described here: API / webservice documentation (log in to see this link).
- locationName: The node of the Ocean Networks Canada observatory. Each location contains many sites.
- dataQualityComments: In some cases, there are particular quality-related issues that are mentioned here. This is distinct from QAQC information contained in the data structure.
- MobilePositionSensor.name: A cell array of sensor names for mobile position sensors. If not a mobile device, this will be an empty cell string.
- MobilePositionSensor.sensorID: An array of unique identifiers of sensors that provide position data for mobile devices - this data may be used in this data product.
- MobilePositionSensor.deviceID: An array of unique identifiers of sensors that provide position data for mobile devices - this data may be used in this data product.
- MobilePositionSensor.dateFrom: An array of datenums denoting the range of applicability of each mobile position sensor - this data may be used in this data product.
- MobilePositionSensor.dateTo: An array of datenums denoting the range of applicability of each mobile position sensor - this data may be used in this data product.
- MobilePositionSensor.typeName: A cell array of sensor names for mobile position sensors. If not a mobile device, this will be an empty cell string. One of: Latitude, Longitude, Depth, COMPASS_SENSOR, Pitch, Roll.
- MobilePositionSensor.offset: An array of offsets between the mobile position sensors' values and the position of the device (for instance, if cabled profiler has a depth sensor that is 1.2 m above the device, the offset will be -1.2m).
- MobilePositionSensor.sensorTypeID: An array of unique identifiers for the sensor type.
- MobilePositionSensor.correctedSensorID: An array of unique identifiers of sensors that provide corrected mobile positioning data. This is generally used for profiling deployments where the latency is corrected for: CTD casts primarily.
- deploymentDateFrom: The date of the deployment on which the data was acquired.
- deploymentDateTo: The date of the end of the deployment on which the data was acquired (may be omitted if still deployed).
- samplingPeriod: Sampling rate of the instrument in seconds (maybe omitted on some devices that have no scalar sensors).
- samplingPeriodDateFrom: matlab datenum of the start of the corresponding sample period (maybe omitted on some devices that have no scalar sensors).
- samplingPeriodDateTo: matlab datenum of the end of the corresponding sample period (maybe omitted on some devices that have no scalar sensors).
- searchID: unique number tracking this search request (not normally included).
- Attribution: A structure array with Attribution information, ordered by importance and date. For internal users, go to the Network Console to configure the attributions. If an organization has more than one role it will be collated. If there are gaps in the date ranges, they are filled in with the default Ocean Networks Canada citation. If the "Attribution Required?" field is set to "No" on the Network Console then the citation will not appear. For data products with a attribution (except MAT files) and for users making products from a MAT file, if the special attribution is blank/null, then the company default attribution will be used and if it is also blank/null, then the final attribution will consist of the organization name and role: "Ocean Networks Canada (Owner, Collaborator)". Here are the fields:
- acknowledgement: the acknowledgement text, note that if the special acknowledgement blank/null, the default acknowledgement is used.
- startDate: datenum format
- endDate: datenum format
Oceans 2.0 API filter:
Using the MAT file
Unlike the .DT4 files, mat files cannot be visualized directly by commercial software. Instead, these files are intended for analysis using MATLAB. In the data structure for each channel, is the vals field that contains the raw A/D counts which can be easily converted into target or volume backscatter strength (via a log transform), depending on what information is required. The target strength is the strength of reverberation from single, large targets such as fish, while the volume backscatter strength is for clouds of scatterers such as plankton. Please consult a good textbook on underwater acoustics for more information, for example: H. Medwin & C. S. Clay, Fundamentals of Acoustical Oceanography (Academic, Boston, 1998). Since amount of data is massive, it would not be practical to include calculated values for all three types of data. However, any MATLAB user can quickly calculate the target or volume backscatter strength from the A/D counts. Here is some example MATLAB code that calculates and plots the data for channel 2: