24

ASL Acoustic Profiler Time Series

Data products for ASL Environmental Sciences echosounders are described here. Currently, three echosounder types are supported: the Zooplankton Acoustic Profiler (ZAP), the Acoustic Water Column Profiler (AWCP) and the Acoustic Zooplankton and Acoustic Zooplankton Fish Profiler (AZFP). The Shallow Water Ice Profiler (SWIP) is a similar instrument, however it only reports ice thickness scalar data, not backscatter, therefore its data is available via scalar data products. Echosounder backscatter data is numerically intense, generating some of the largest data sets at ONC. Because of this, ASL echosounder data products for extended time ranges will be slow to generate and download. In addition, ASL echosounders also produce some ancillary scalar data, such as temperature, pitch, roll, which are available independently as scalar data products.

ONC has worked closely with ASL Environmental Sciences, providing a testbed for the development of their echosounders, especially for their calibration. Users may notice that the ZAP is the simplest of the echosounders, while the AZFP is the most advanced. This is reflected in the MAT file product as MAT files from the ZAPs have a number of empty fields. In order to create a MAT file product that works with as many echosounders as possible, including multi-frequency AWCPs and AZFPs, the format and layout of the MAT file is more complex then the original ZAP mat file format available on the VENUS data download page. Users of ZAP data from the VENUS data download page will find more information on the more generic ASL MAT file format below, including a matlab script that can be used to convert MAT files into the VENUS ZAP format. The script is also useful as a guide for users to update their code to the new file format. Once updated, users can process ASL data from a combination of ZAP, AWCP and AZFP echosounder over a long time range. (ZAPs are slowly being phased out in favour of AZFPs.)

Manufacturer's documentation and software (if available):

ASL ZAP Documentation

ASL AWCP Documentation

ASL AZFP Documentation

AZFP website: https://aslenv.com/azfp.html

Oceans 2.0 API filter: dataProductCode=AAPTS

Revision History

1. 20110302: Beta MAT product released
2. 20141028: Major revision: support AZFP and AWCPs, integrate options, process requests from all networks, improve performance, standardize MAT files, add plots
3. 20150504: Major revision: add support for ZAPs, add sun elevation graph

Data Product Options

Formats

This data is available in MAT, PNG and PDF formats. Content descriptions and examples are provided below. When a driver/device restart is detected, the configuration is checked and if the configuration changed, all data products will break and a new file will be started (configuration breaks are also noted in the search status and logs). For all non-averaged data products and daily plots, a new file is started at the start of each day (in addition to configuration breaks). For ensemble-averaged MAT files and multi-day plots, the files can extend over multiple days, while file breaks will still occur for configuration changes and if the files get too big (MAT files will break if they occupy more than 1 GB memory or about ~400 MB on disk). 

Calibrated data are represented in volume backscatter strength in units of dB re 1 m^-1. Without calibration, the data is in units of raw counts which is proportional to the pressure amplitude. When plotting raw data, the data is still plotted in a dB scale, but instead of an absolute measure in volume backscatter strength, the plot uses a dB scale relative to the full scale of the device, where 0 dB represents the maximum value, normally 65535 raw counts for 16 bit data, and the lowest value is the 20*log10 value of the smallest value the A/D converter will produce (about -48 dB for ZAPs, about -96 dB for AWCPs and about -120 dB for AZFPs).

For multi-frequency echosounders with separately mounted transducers, we account for the height variances with the device attribute Echosounder_TransducerHeight, which is utilized in both the plots and presented in the Meta structure in the MAT files.

Data files: MAT

MAT files (v7) can be opened using MathWorks MATLAB 7.0 or later. The file contains four structures: Meta, Data, Config, and Units. The structures are the same for AWCP, AZFP or ZAP ASL echosounders. VENUS data download search requests for AZFP and AWCP echosounders will also use this format, while ZAP search requests from VENUS data download will produce a different format. Data Search / Oceans 2.0 ZAP MAT files can be converted to the VENUS ZAP format by a conversion script that is provided below. Screenshots below capture what the MAT file will look like loaded in MATLAB's workspace. 

Not shown above:

Meta.transducerHeight - This is an offset height (m) relative to the instruments' base level, used to account for the varying heights of multi-frequency, separately mounted tranducers. By ONC convention the site depth is the depth the instrument platform floor, while the siteDevice offset is the depth change from the platform to the instrument, but in this case that offset will be measured to the transducer mounting base.

Data: structure containing the AWCP data, having the following fields.

• time: vector, timestamp in datenum format (obtained from time the reading reached the shore station, or if the ping averaging option was selected, these time stamps will be resampled).
• temperature: vector, temperature time-series (will be filled with NaNs if not available). Calibration formula:
  vin = tempRawCounts * 2.5 / 65535; 
  R = (ka + kb * vin) / (kc - vin); 
  temperature = (1/(A + B * log(R) + C * (log(R)^3))) - 272.15;
• pressure: vector, pressure time-series (will be filled with NaNs if not available)
• tiltX: vector, tilt X-direction time-series (will be filled with NaNs if not available). Calibration formula: 
  X_a * txc + X_b * txc + X_c * txc + X_d, where txc is the raw tilt counts in the x direction
• tiltY: vector, tilt Y-direction time-series (will be filled with NaNs if not available).  Calibration formula: 
  Y_a * txc + Y_b * txc + Y_c * txc + Y_d, where txc is the raw tilt counts in the x direction
• voltage: vector, battery voltage time-series (will be filled with NaNs if not available)
• timeInstrumentClock: vector, instrument clock time-series
• ProfileNum: vector, profile number
• error: vector, error code
• nPingsAquired: vector, number of pings acquired for profile
• pingFirst: vector, index of first ping acquired in profile
• pingLast: vector, index of last ping acquired in profile
• profileStatus: vector, profile status code
• overrun: vector, (we think) this indicates if the data hit the limit of the A/D converter.
• Channel: structure (one for each channel, so access is by Data.channel(3) for example), with the following fields:
  • rangeToBinCentre: vector of length of the number of bins. The range to the centre of each bin is calculated from two-way travel time, accounting for sound speed, lockout distance and one half of the pulse length.
  • profileData: 2D matrix (time, rangeToBinCentre) of the acoustic data. The acoustic data can be raw counts from the echosounder's Analogue to Digital converter (A/D), or if calibrated, it will be volume backscatter data in units of dB re 1 m^-1. See the Unit structure or the isCalibrated flag (below) in order to tell the two apart. Here is the calibration formula applied to convert raw profile data to calibrated profile data for AWCPs and AZFPs:
    

    N = (log10(Data.Channel(i).profileData) - 2.5) * 524280 * Config.Calibration.DS(i);

    Data.Channel(i).profileData = Config.Calibration.EL(i) - 2.5/Config.Calibration.DS(i) + ...
    N/(26214*Config.Calibration.DS(i)) - Config.Calibration.TVR(i) - 20*log10(Config.Calibration.VTX0(i)) + 20*log10(R) + ...
    2*Config.Calibration.acousticAttenuation(i) * R - ...
    10*log10( (1/2) * Config.speedOfSound * (Config.pulseLength(i)*1e-6) * Config.Calibration.BP(i) );

    
    Here is the calibration formula for ZAPs:

    Data.Channel(i).profileData = -Config.Calibration.OCV(i) - Config.Calibration.sourceLevel(i) - 20*log10(Config.Calibration.Q(i)) + 40*log10(R) +...
    2*Config.Calibration.acousticAttenuation(i) * R -...
    10*log10( (1/2) * Config.speedOfSound * (Config.pulseLength(i)*1e-6) * Config.Calibration.BP(i) * R.^2 ) +...
    20*log10(Data.Channel(i).profileData) - repmat(Config.Calibration.zapInterpTVG(i,:), size(Data.Channel(i).profileData, 1), 1);

    
    where i is the channel number and R is the rangeToBinCentre (replicated into a matrix). The calibration procedure also accounts for the gain and time-varying-gain applied. In the case of the ZAP echosounder, a noise reduction algorithm is also applied (contact us for more details). 
     
  • rawData: 2D matrix (time, rangeToBinCentre) of the acoustic data. When non-calibrated data is requested for ZAP echosounders only, this matrix is provided as the most raw data. In that case, profileData represents the noise-reduced raw data (contact us for more details). 
     
  • isCalibrated: boolean indicates if the data is calibrated and is volume backscatter data in units of dB re 1 m^-1.

• • invalidProfileData: logical vector, custom added field by our code that indicates if the raw data was truncated before the expected number of range bins was returned. Invalid data are filled with NaNs. We have seen partial data in truncated pings, which could be extracted, however, this data does not appear to be reliable. If you wish to see this data, contact us and we can provide it. 

Config: structure containing instrument configuration details. For details about the configuration parameters, refer to the manufacturer documentation. This structure contains a conglomeration of parameters from all 3 types. All parameters are the original instrument settings as read from the raw data except otherwise noted (in particular, Calibration, speedOfSpeed, orientation and ProcessingOptions are applied in postprocessing to calibrate the data)

• echosounderType: string, name of the echosounder type (used on the plot).
• nChannels: number of channels (in case instrument is capable of having multiple frequencies)
• serialNum: serial number of device
• profileInterval: number of seconds per profile (or burst interval, reflects onboard ensemble averaging for AWCP / AZFP, otherwise equal to data rating or ping interval)
• digitizationRate: rate at which received signal is digitized, per channel
• lockoutSamples: number of samples to ignore
• nBins: number of bins in profile, per channel
• nSamplesPerBin: number of samples used to calculate amplitude in each bin, per channel. This is known as Range Averaging. For AZFPs and AWCPs, this is a configurable parameter, for ZAPs it was fixed.
• nSamples: number of samples in profile, per channel. This is the number of raw samples: nBins * nSamplesPerBin or maxRange * digitizationRange * 2 / speedOfSound, where maxRange is not the same as rangeMaximum below, but is max(Data.Channel(1).rangeToBinCentre) + binRange/2 
• pingsPerProfle: number of pings to average for profile, per channel (onboard ensemble averaging only)
• pingPeriod: number of seconds between pings
• sumSquaresFlag: flag to indicate if sum of squares values calculated onboard AWCPs only and included in the raw data (onboard ensemble averaging only)
• averagedPings: flag to indicate if averaging was done onboard AZFPs only (onboard ensemble averaging only, probably replaced the sumSquaresFlag) 
• phase: phase used to acquire profile (AWCP/AZFP only, not used in any processing or display)
• gain: gain setting, per channel (affects time-varying-gain on AWCP and ZAP)
• pulseLength: pulse length, per channel
• boardNumber: board number, per channel

These values may not be read from the raw data:

• speedOfSpeed: sound speed used for calculation of range (single value determined from median conditions in the water column for device's deployment location, in units of m/s), used in post-processing.
• orientation: 'Up' is the normal case for all echosounders mounted on the seabed, facing up towards the sea surface, while 'Down' is the case for echosounders mounted on ships, buoys, etc. We use this flag for plotting purposes. The y-axis in the time-series 'echogram' plots will be calculated as follows: for 'Up': binDepth = Meta.depth - Data.channel.rangeToBinCentre, for 'Down': binDepth = Meta.depth - Data.channel.rangeToBinCentre, unless Meta.depth is NaN. In that case, we plot the echosounder's range as the y-axis. (Meta.Depth == NaN means the echosounder is mobile. In the future we may produce a plot that accounts for mobile echosounders.) (Available late November, 2014.)
• Calibration: structure containing calibration data stored in device attributes. For AZFP and AWCPs, these values originate from configuration text (.cfg) files that ship with the instruments. For ZAPs, these values originate from ONC calibration reports. See the instrument documentation pages for more details, we can also provide the formulas by which these values are applied. The Calibration structure includes:
  • tilt sensor calibration coefficients (fields starting with X or Y)
  • pressure sensor calibration co-efficients (fields starting with k, plus A, B, C)
  • backscatter calibration parameters, fields: TVR, VTX0, EL, DS, BP, AcousticAttenuation (attenuation is a single value determined from median conditions in water column for device's deployment location, in units of dB/m). For ZAP: OCV, Q, sourceLevel, zapInterpTVG, zapNoise, zapGainMultiplier
• frequency: frequency of echosounder, per channel (AWCP/AZFP read from data, ZAP stored attribute)
• binRange: two-way distance spanned by each bin (calculated from digitizationRate, speedOfSpeed, pulseLength)
• lockoutDistance: blanking distance (based on number of lockout samples), per channel
• rangeMaximum: maximum range from which samples are digitized, per channel. This is useful to indicate the last or furthest possible contribution to the data, which is distinct from the maximum value of Data.channel.rangeToBinCentre. For rangeMaximum, the value is calculated as the number of bins times the bin range, plus the pulse duration times sound speed divided by 2; in other words, to the end of the last bin and to the end of transmit pulse. 
• PostprocessingOptions: structure containing the post-processing options
  • finalEnsemblePeriod: final ensemble period. For non-averaged mat files, this will be Config.profileInterval, for plots and averaged mat files, this is the final ensemble averaging period
  • ensemblePeriodRequested: ensemble averaging option selected by the user
  • calibrationRequested: calibration averaging option select by the user, calibration may not be available, see Data.Channel.isCalibrated. 1 for yes, 0 for no
  • sunElevationRequested: show the sun elevation graph? 1 for yes, 0 for no (for plotting products only, for MAT files, this will be empty)

Units: structure containing unit of measure for fields in structures above. For instance, units.frequency='kHz'. Note that channel specfic data units: profileData, rangeToBinCentre are stored in Units.channel(1).

Oceans 2.0 API filter: extension=mat

ASL Profiler Time Series MAT file Conversion Script

This script will convert MAT files to the VENUS ZAP MAT file format. It works on data loaded in the workspace and is intended to be used as basis for users to adapt their code and analysis to the new format, which will be the standard format into the future. Download the script file and open in matlab, press the 'run' button, the script will open a window to select an ASL Profiler MAT file to process, after loading the ASL Profiler MAT file, the workspace will be populated with the familar ZAP MAT file structures: zapdata, config, units, meta, while the ASL MAT file structures will remain (ASL MAT file structures follow the MATLAB convention of naming structures with upper case letters).

convertASLmatToZAPmat.m

Plots: PNG and PDF

There are two formats of plots available: PNG or PDF. A PDF plot file can contain multiple plots as separate pages, and the graphics are vector images, which are better for printing or viewing at high resolution.  The PNG format is a single plot in a raster image which is good for quick viewing and sharing. The data and appearance of the two plot formats are the same. These plots are also known as echograms, they plot echo intensity (backscatter) vs time and depth, and are basically what you would see as a sonar operator.

Oceans 2.0 API filter: extension={png,pdf}

Plots: Daily vs Multi-day plots

There are two variations of plots, depending on the ensemble period option selected. Daily plots are generated with the default, no averaging, option. Daily plots will show a maximum of one day of data. For all plots, the data has to be resampled so that each pixel in the PNG is an ensemble period, otherwise rendering the image will alias the data; so in spite of selecting the no-averaging option, some resampling may happen. Also, nominal linear image anti-aliasing routines aren't appropriate here as we have logarithmic data. The minimum ensemble period for a one day plot is ~30 seconds. If you select less than one day, you can zoom in and get higher temporal resolution. Here are some examples of daily plots: first is a ZAP calibrated daily echogram with the sun elevation plot; second is a ZAP daily echogram, non-calibrated without the sun elevation plot, with a configuration change part way through the day; third is an AZFP calibration daily echogram without the sun elevation plot.

If an ensemble period is selected by the user, the plots will be multi-day plots and only one plot will be generated over the search time range (excluding configuration changes). Ensemble averaging will be applied as selected. If the selected ensemble period is not high enough to prevent aliasing and distortion, the ensemble period will be increased automatically. Below are examples of multi-day plots from a ZAP and an AZFP with ensemble averaging set to 10-minutes, which is easily long enough to avoid aliasing and distortion. If plots extend over gaps in the data, users will see the gaps represented by white space.

Plots: Single vs. Multiple Channels

If the echosounder has multiple channels, as in the examples above, each channel will be plotted as a subplot, with independent axis and limits. (Axis limits are set from fixed intervals, e.g. every 20 dB, to facilitate inter-plot comparison.) In addition, if the device has precisely 3 channels, an additional RGB composite plot will be shown. For this subplot, each channel's data scaled from 0 to 1, each channel is then represented by a primary colour, and the colours are combined to form an image. In this way, users can see composite details: various targets will appear as different colours, depending on their relative target strength as a function of frequency. This is useful for differentiating the targets between fish, zooplankton, bubbles, whales, etc.

Plots: Mobile Depth

Here is an example of single channel plot, from a mobile AWCP echosounder on the vertical profile system where the y axis is not depth, but is instead echosounder range. We can't use depth as it is varying throughout the data. Future improvements maybe to register the pings with the varying depth, so that the y axis can be depth. 

Discussion

To comment on this product, click Add Comment below.