BioSonics Time Series
Data products for BioSonics Echosounders are described here. Please note, new data products become available every day shortly after the log file is closed and processed at midnight UTC (4 or 5 PM Pacific time). The conversion from log to DT4 formats takes about 20 minutes for one day of data, as does the conversion from DT4 to MAT. Once the data is converted, the DT4 and MAT files are archived for quick retrieval, (not all MAT files are currently archived, so conversion is done on demand).
Revision History
- 2011XXXX: Initial data products released in manufacturer DT4 and MATLab mat formats.
Oceans 2.0 API
Formats
This data is available in DT4 and MAT formats. Content descriptions and example files are provided below.
To produce the file, the following requirements apply:
- A new DT4 file is started at the beginning of each day (midnight UTC exactly), or when the driver is restarted (this should account for configuration changes, site changes, etc).
- The instrument date/time field is replaced using the NEPTUNE timestamp at the beginning of the log file (since this timestamp is more accurate than the instrument clock).
- MAT files are made from DT4 files. MAT files are made on the hour every hour, except when the data is truncated, then small files are created. The date-from and date-to of a DT4 file and the MAT files created from it may not agree if the DT4 file doesn't contain data for it's full time range.
- Although each channel/frequency pings simultaneously, simultaneous ping packets will have slightly different time stamps, usually separated by 100 milliseconds. This means simultaneous ping packets may end up in different DT4 files if they occur on either side of midnight. The only way to sort this out is to look at the times in the MAT files.
DT4
This format is specific to the manufacturer, and consists of sections called tuples (e.g., channel descriptor, time and ping tuples). When using BioSonics data acquisition software, data is normally stored in this way. Although we use custom-built drivers to communicate with our instruments, we can use the raw data in the log file to produce the DT4 file which can be interpreted by BioSonics post-processing software (most recent version available from BioSonics website) in play-back mode, and commercial software like EchoView.
A new file is generated each day, and whenever the driver is restarted. The reference time in the time tuple and the ping time in the ping tuples correspond to the times the measurements were received at the NEPTUNE Canada shore station (in place of the internal instrument clock values).
The format and tuple contents are further described in the manufacturer's documentation. However, there are two tuples that are not described in this document, but are regularly included in DT4s.
pulse tuple (0X0013) - describes the pulse type of the data for each configuration
06 00 13 00 00 00 01 00 00 00 0C 00 indicates that configuration #1 is raw data pulse type 06 00 13 00 00 00 02 00 00 00 0C 00 indicates that configuration #2 is raw data pulse type 06 00 13 00 00 00 03 00 00 00 0C 00 indicates that configuration #3 is raw data pulse type
marker tuple (0X0010 and 0X0020) - used to indicate the start and end of the time & ping tuples
04 00 10 00 01 00 00 00 0A 00 signifies start of time/ping data 04 00 10 00 02 00 00 00 0A 00 signifies end of time/ping data
MAT
MAT files (v7.3) can be opened using MathWorks MATLAB 7.3 (R2006b) or later. These files use 64-bit indexing, but will work on a 32-bit operating system (will be slower). A new file a created every hour from the start of the parent DT4 file(s). One hour files were selected so that the memory requirements are not too onerous. Opening a one-hour BioSonics MAT file in MATLAB will require about 500 MB of contiguous available memory. Various plotting routines and analysis may cause MATLAB to use up to 2 GB of memory.
Each MAT file contains the following structures: meta, data, units. Entries in italic not applicable to the type of BioSonics Echosounders deployed, but are standard data fields in the DT4 file specification.
meta: structure containing the following metadata fields, generated entirely from NEPTUNE's database.
deviceID: A unique identifier to represent the instrument within the NEPTUNE Canada network.
creationDate:Date and time (using ISO8601 format) that the data product was produced. This is a valuable indicator for comparing to other revisions of the same data product.
deviceName: A name given to the instrument.
deviceCode: A unique string for the instrument which is used to generate data product filenames.
deviceCategory: Device category to list under data search ('Echosounder').
lat: Obtained at time of deployment.
lon: Obtained at time of deployment.
depth: Obtained at time of deployment.
deviceHeading: Obtained at time of deployment.
siteName: Name corresponding to its latitude, longitude, depth position.
locationName: The node of the NEPTUNE Canada observatory. Each location contains many sites.
deploymentDateFrom: The date of the deployment on which the data was acquired
deploymentDateTo: The date of the end of the deployment on which the data was acquired
samplingPeriod: Sampling rate of the instrument in seconds.
searchID: The search ID from the data search details structure used to find this data (not applicable to post-processed MAT files)
data: structured array containing the BioSonics data (one structure per configuration in cycle), having the following fields which pertain to the dt4 contents (for details, refer to the manufacturer manual).
name: serial number
frequency: transducer frequency (Hz)
time: datenum vector, UTC timestamp (time source is NEPTUNE Canada shore station)
range: vector of distance to each bin
pingnum: ping number
vals: amplitude of return signal (A/D Counts - main data matrix - data type uint32)
nsamps: number of samples in the ping tuples (read from the ping tuple)
env: structure containing environmental configuration values
- absorb: configuration absorption coefficient (dB/m - specific to each channel/frequency)
- sv: configuration sound velocity (m/s)
- temperature: configuration temperature (degrees C)
- salinity: configuration salinity (ppt)
- power: transmit power reduction factor
- numChan: total number of frequencies/channels/configurations in the dt4 file
snd: structure containing details of the channel descriptor tuple
- address: configuration/channel number
- npings: total number of pings in the dt4
- npingsExtracted: total number of pings extracted from the dt4
- sampperping: samples per ping (this is usually equal to max(data.nsamps) )
- sampperiod: time between successive samples in a ping (microseconds)
- pingperiod: number of seconds between successive pings
- pulselen: duration of transmitted pulse (milliseconds)
- blank: initial blanking (number of samples skipped before data recording begins for each ping)
- threshold: data collection threshold (dB)
- rxee: receiver EEPROM image structure (can ignore split-beam options since that feature is not included in this instrument)
- ssn: transducer SN
- calDateNum: last calibration date (Matlab datenum)
- calDateStr: last calibration date (datestring)
- calTech: initials of technician performing calibration
- sl: source level (0.1 dB)
- rs: receiver sensitivity for main element (0.1 dB)
- rsw: receiver sensitivity for wide-beam (0.1 dB)
- pdpy: sign of split beam y-axis (zero=positive, nonzero=negative)
- pdpx: sign of split beam x-axis (zero=positive, nonzero=negative)
- noiseFloor: noise floor of transducer (AD Counts) due to system noise
- transducerType: transducer hardware type (0 for single-beam, 3 for dual-beam, 4 for split-beam)
- frequency: transducer frequency (Hz)
- pdy: absolute value of the split-beam y-axis element separation (0.01 mm)
- pdx: absolute value of the split-beam x-axis element separation (0.01 mm)
- phpy: split-beam y-axis element polarity
- phpx: split-beam x-axis element polarity
- aoy: split-beam y-axis angle offset (0.01 degrees)
- aox: split-beam x-axis angle offset (0.01 degrees)
- bwy: y-axis -3dB one-way beam width for the main element (0.1 degrees)
- bwx: x-axis -3dB one-way beam width for the main element (0.1 degrees)
- sampleRate: sampling rate of the A/D converter (Hz), usually 41667.
- bwwwy: y-axis -3dB one-way beam width for wide-beam or phase elements (0.1 degrees)
- bwwx: x-axis -3dB one-way beam width for wide-beam or phase elements (0.1 degrees)
- phy: split-beam y-axis phase aperture (0.1 degrees)
- phx: split-beam x-axis phase aperture (0.1 degrees)
- ccor: calibration correction
opmode: operating mode
units: structure containing unit of measure for fields in structures above. For instance, units.lat='degrees N'.
Using the MAT file
Unlike the .DT4 files, mat files cannot be visualized directly by commercial software. Instead, these files are intended for analysis using MATLAB. In the data structure for each channel, is the vals field that contains the raw A/D counts which can be easily converted into target or volume backscatter strength (via a log transform), depending on what information is required. The target strength is the strength of reverberation from single, large targets such as fish, while the volume backscatter strength is for clouds of scatterers such as plankton. Please consult a good textbook on underwater acoustics for more information, for example: H. Medwin & C. S. Clay, Fundamentals of Acoustical Oceanography (Academic, Boston, 1998). Since amount of data is massive, it would not be practical to include calculated values for all three types of data. However, any MATLAB user can quickly calculate the target or volume backscatter strength from the A/D counts. Here is some example MATLAB code that calculates and plots the data for channel 2:
%load a biosonic mat file either drag and drop or use something like:
%uiload
chn = 2;
vals = double(data(1,chn).vals);
%plot the A/D vals
xStartDay = floor(min(data(1,chn).time));
x = (data(1,chn).time - floor(min(data(1,chn).time)))*24*60;
y = fliplr(data(1,chn).range.');
figure;
imagesc(x,y,flipud(vals))
set(gca, 'ydir', 'normal')
ylabel('Range (m)')
xlabel(['Minutes since ' datestr(xStartDay,'dd-mmm-yyyy') ' 00:00:00'])
title('A/D values')
%Calculate the receive pressure - just for fun
p = vals/10^(data(1,chn).snd.rxee.rs/200);
%Calculate and plot the volume backscatter strength
a = data(1,chn).env.absorb; %calculated from environmental conditions in dB/m for each channel/frequency
psi = data(1,chn).snd.rxee.bwy/20*data(1,chn).snd.rxee.bwx/20*(10^-3.16);
Sv = 20*log10(vals) -...
(data(1,chn).snd.rxee.sl + data(1,chn).snd.rxee.rs + data(1,chn).env.power)/10 +...
repmat(20*log10(data(1,chn).range) + 2*a*data(1,chn).range, 1, size(vals,2)) -...
10*log10(data(1,chn).env.sv*data(1,chn).snd.pulselen/1000*psi/2) +...
data(1,chn).snd.ccor/100;
figure;
imagesc(x,y,flipud(Sv))
set(gca, 'ydir', 'normal')
ylabel('Range (m)')
xlabel(['Minutes since ' datestr(xStartDay,'dd-mmm-yyyy') ' 00:00:00'])
title('Volume Backscatter Strength')
%Calculate and plot the target strength
TS = 20*log10(vals) -...
(data(1,chn).snd.rxee.sl + data(1,chn).snd.rxee.rs + data(1,chn).env.power)/10 +...
repmat(40*log10(data(1,chn).range) + 2*a*data(1,chn).range, 1, size(data(1,chn).vals,2)) +...
data(1,chn).snd.ccor/100;
figure;
imagesc(x,y,flipud(TS))
set(gca, 'ydir', 'normal')
ylabel('Range (m)')
xlabel(['Minutes since ' datestr(xStartDay,'dd-mmm-yyyy') ' 00:00:00'])
title('Target Strength')