You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 48 Next »

Nortek Time Series

Data files for Nortek devices (ADCPs, current meters, velocimeters) are described here. Daily current plots are also available for some Nortek devices. Parsed scalar ancillary data (e.g., temperature, pitch, roll) are also available as time series scalar data and time series scalar plots. 

Revision History

  • 20150226: Change range to be to the beam centre, add bin mapping to correct velocities and range for tilt. Add all other Nortek raw binary formats (VEC, VNO, AQD, WPR).
  • 20150202: Heading correction rework (affecting ENU / uvw velocities): improve handing of mobile sensors, autonomous deployments and better documentation of processing was done to the data.
  • 20111019: Initial NetCDF, CF1.5 compliant product released.
  • 20110302: Correction to algorithm for obtaining ENU (with respect to true North velocities). If instrument is mobile, a combination of instrument compass and local magnetic declination is used. If instrument is fixed position, the orientation determined at instrument deployment is used.
  • 20110110: Initial MAT product released
  • 20101130: Initial PRF product released

Formats

This data is available in manufacturers' raw binary data files (PRF / AQD / VEC / VNO / WPR), and parsed and processed time series data files (MAT and NETCDF). Content descriptions are provided below.

To produce these files, the following requirements apply:

  • A new file is started at the start of each day, when the maximum records per file is exceeded (86400 for MAT and NETCDF, 86400 for PRF / AQD / VEC / VNO / WPR, or when the driver is restarted (this should account for configuration changes, site changes, etc).
  • Only records with valid checksums are included.
  • Only records that have a valid Nortek data structure ID, including configuration data structures and structures that may not normally be associated with file extension (Nortek software appears to handle this ok. We don't filter out valid data structures as firmware updates may cause a structure to be used by a different device). List of valid data structures was taken from the system integrator manual, version Dec. 2014.
  • The instrument date/time field is replaced by the ONC timestamp at the beginning of the log file (since this timestamp is more accurate than the instrument clock).

All Nortek devices are sensitive to orientation. Current meters and velocimeters' data are parsed into scalar data, and part of that process includes a rotation from instrument or Earth relative velocities to true East, North and Up velocities. Data from the manufacturers' raw binary files (PRF / AQD / VEC / VNO / WPR) is as is: no rotation or processing is done. Users should use the appropriate post-processing software to rotate the data to correct for orientation. The MAT and NETCDF processed data products described further below are generated correction for orientation that is described in detail below.

Raw Binary Data Files: PRF / AQD / VEC / VNO / WPR

These very similar binary formats are specific to Nortek acoustic devices. When using Nortek data acquisition software, data is normally stored in this way. Although we use custom-built drivers to communicate with our instruments, we can use the raw data in the log file to produce these files. The aforementioned requirements apply, as well as:

  • After replacing the instrument date/time field with the ONC timestamp, the checksum is recalculated.
  • Instrument hardware, head and user configuration strings are inserted as the first three lines of the each raw binary file and important parameters are parsed into MAT file. However, any data existing previous to accompanying configuration details being saved in the database cannot be interpreted (earlier than Fall 2009). We are actively back-filling the configurations, contact us or press the help button if your data product request indicates that the configuration does not exist, even though there is data available.

VEC is the format for Nortek Vector Current Meters, VNO is the format for Nortek Vectrino Current Meters, AQD is the format for Nortek Aquadopp Current Meters, PRF is the format for Nortek ADCPs, both the Aquadopp Current Profiler and the Aquadopp-HR Current Profiler, WPR is the format for Nortek AWAC (measures currents and waves, see Nortek's description here). Go to each of those links for the appropriate manufacturer's software for each. Each type of Nortek device has it's own software, however, they all have the same data conversion function that appears to read all of the above formats and data structures (it's best to use the appropriate software). In the appropriate Nortek device interface software, go to the 'Deployment' menu, select 'Data Conversion...', then 'Add file' and then press the arrow. Text file conversions of the raw binary data will be produced. Nortek also offers post-processing software, such as Surge, Prof2NDP, bin2mat, etc., that will also read these formats, see here for downloads (some require license): http://www.nortek-as.com/en/support/software

These formats are further described in the manufacturer's System Integrator Guide.

Example: BH_POD2_AD2M_20101018T160041.704.prf

Errors and Warnings in Nortek Software

When reading and converting the raw binary data files with Nortek Software, users may encounter some warnings. In our testing, we've come across the following warnings and messages:

In the .ssl file that was was generated by Aquadopp from the AQD file, we saw some warnings about out of range pitch and roll sensor data. This could be caused by a bad configuration or orientation, most likely test data, or this can occur if the orientation of the device is too extreme, as could happen if the device was deployed to a mooring or mobile deployment. In the case of the latter, users may want to ignore or mask out this data.

In the Vector software, we often see this warning. It will create a small subset VEC file that appears to duplicate data that was processed normally. The processed data appears to be ok; it is not out of order or otherwise scrambled.

Contact us if you have any questions about these errors.

Processed Time Series Data Files: MAT / NETCDF

Processed time series data files are available for Nortek ADCP-type devices: Aquadopp Current Profiler, Aquadopp-HR Current Profiler. The Nortek current meter type devices are parsed as scalar sensors. The processed data for those devices are available in generic time series scalar data and time series scalar plot data products, and from other scalar data portals such as Plotting Utility.

Data Product Options: Ensemble averaging
Unable to render {include} The included page could not be found.


Unable to render {include} The included page could not be found.
MAT

MAT files (v7) can be opened using MathWorks MATLAB 7.0 or later. The file contains four structures: meta, data, config, and units. 

Meta: a structure array containing the following metadata fields:

  • deviceID: A unique identifier to represent the instrument within the Ocean Networks Canada data management and archiving system.
  • creationDate:Date and time (using ISO8601 format) that the data product was produced. This is a valuable indicator for comparing to other revisions of the same data product.
  • deviceName: A name given to the instrument.
  • deviceCode: A unique string for the instrument which is used to generate data product filenames.
  • deviceCategory: Device category to list under data search ('Echosounder').
  • deviceCategoryCode: Code representing the device category. Used for accessing webservices, as described here: API / webservice documentation (log in to see this link).
  • lat: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site latitude and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure..
  • lon: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site longitude and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • depth: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site depth and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • deviceHeading: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site heading and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • devicePitch: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site pitch and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • deviceRoll: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site roll and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • siteName: Name corresponding to its latitude, longitude, depth position.
  • locationName: The node of the Ocean Networks Canada observatory. Each location contains many sites.
  • stationCode: Code representing the station or site. Used for accessing webservices, as described here: API / webservice documentation (log in to see this link).
  • dataQualityComments: In some cases, there are particular quality-related issues that are mentioned here.
  • MobilePositionSensor: A structure with information about sensors that provide additional scalar data on positioning and attitude (latitude, longitidue, depth below sea surface, heading, pitch, yaw, etc).

    • name: A cell array of sensor names for mobile position sensors. If not a mobile device, this will be an empty cell string.
    • sensorID: An array of unique identifiers of sensors that provide position data for mobile devices - this data may be used in this data product.
    • deviceID: An array of unique identifiers of devices that provide position data for mobile devices - this data may be used in this data product.
    • dateFrom: An array of datenums denoting the range of applicability of each mobile position sensor - this data may be used in this data product.
    • dateTo: An array of datenums denoting the range of applicability of each mobile position sensor - this data may be used in this data product.
    • typeName: A cell array of sensor names for mobile position sensors. If not a mobile device, this will be an empty cell string. One of: Latitude, Longitude, Depth, COMPASS_SENSOR, Pitch, Roll.
    • offset: An array of offsets between the mobile position sensors' values and the position of the device (for instance, if cabled profiler has a depth sensor that is 1.2 m above the device, the offset will be -1.2m).
    • sensorTypeID: An array of unique identifiers for the sensor type.
    • correctedSensorID: An array of unique identifiers of sensors that provide corrected mobile positioning data. This is generally used for profiling deployments where the latency is corrected for: CTD casts primarily.
  • deploymentDateFrom: The date of the deployment on which the data was acquired.
  • deploymentDateTo: The date of the end of the deployment on which the data was acquired (will be NaN if still deployed).
  • samplingPeriod: Sample period / data rating of the device in seconds, this is the sample period that controls the polling or reporting rate of the device (some parsed scalar sensors may report faster, some devices report in bursts) (may be omitted for some data products).
  • samplingPeriodDateFrom: matlab datenum of the start of the corresponding sample period (may be omitted for some data products).
  • samplingPeriodDateTo: matlab datenum of the end of the corresponding sample period (may be omitted for some data products).
  • sampleSize: the number of readings per sample period, normally 1, except for instruments that report in bursts. Will be zero for intermittent devices (may be omitted for some data products).
  • SamplePeriodSensor: A structure array with an entry for each scalar sensor on the device (even though this metadata is for complex data products that don't use scalar sensors).

    • sp: sample period in seconds (array), unless sensorid is NaN then this is the device sample period
    • dateFrom: array of date from / start date (inclusive) for each sample period in MATLAB datenum format.
    • dateTo: array of date to / end date (exclusive) for each sample period in MATLAB datenum format.
    • sampleSize: the number of readings per sample period (array). Normally 1, except for instruments that report in bursts. Will be zero for intermittent devices.
    • deviceID: array of unique identifiers of devices (should all be the same).
    • sensorID: array of unique identifiers of sensors on this device.
    • isDeviceLevel: flag (logical) that indicates, when true or 1, if the corresponding sample period/size is from the device-level information (i.e. applies to all sensors and the device driver's poll rate).
    • sensorName: the name of the sensor for which the sample period/size applies (much more user friendly than a sensorID).
  • citation: a char array containing the DOI citation text as it appears on the Dataset Landing Page. The citation text is formatted as follows: <Author(s) in alphabetical order>. <Publication Year>. <Title, consisting of Location Name (from searchTreeNodeName or siteName in ONC database) Deployed <Deployment Date (sitedevicedatefrom in ONC database)>. <Repository>. <Persistent Identifier, which is either a DOI URL or the queryPID (search_dtlid in ONC database)>. Accessed Date <query creation date (search.datecreated in ONC database)>
  • Attribution: A structure array with information on any contributors, ordered by importance and date. If an organization has more than one role it will be collated. If there are gaps in the date ranges, they are filled in with the default Ocean Networks Canada citation. If the "Attribution Required?" field is set to "No" on the Network Console then the citation will not appear. Here are the fields:
    • acknowledgement: the acknowledgement text, usually formatted as "<organizationName> (<organizationRole>)", except for when there are no attributions and the default is used (as shown above).
    • startDate: datenum format
    • endDate: datenum format
    • organizationName
    • organizationRole: comma separated list of roles
    • roleComment: primarily for internal use, usually used to reference relevant parts of the data agreement (may not appear)

data: structure containing the ADCP data, having the following fields.

  • time: vector, timestamp in datenum format (obtained from time the reading reached the shore station)
  • range: vector of distance to each bin centre. If bin mapping compensation for tilt is active, this is then a vertical range to the bin centres, see adcp.processingComments to see if bin mapping was applied. (Prior to Feb 26, 2015: vector of distance to start of each bin)
  • cellDepth: vector, starting depth of each cell calculated from range and deployment depth (may be inaccurate if instrument is vertically mobile)
  • u,v,w: 2D matrices, East/North/Up velocities relative to True North, derived using raw velocity matrix, orientation, coordinate system, transformation matrix and one of a) calculated magnetic declination based on time and location (for mobile adcps) or b) meta.deviceHeading (fixed position ADCPs)
  • u_std, v_std, w_std: standard deviations for ensemble averaged u,v,w (averaged products only).
  • pingsPerEnsemble: vector, number of pings that contributed to the ensemble (averaged products only).
  • velocity: 3D matrix, corresponds directly to output of instrument and so depends on configuration coordinate system
  • amplitude: 3D matrix, amplitude time-series for each of the three receivers
  • correlation: 3D matrix, signal correlation coefficient time-series for each of the three receivers (only available for HR-Profilers)
  • compassHeading: vector, magnetic compass heading time-series
  • pitch: vector, pitch time-series
  • roll: vector, roll time-series
  • pressure: vector, pressure time-series
  • temperature: vector, temperature time-series
  • soundSpeed: vector, sound speed time-series (often constant, depending on configuration)
  • voltage: vector, battery voltage time-series
  • error: vector, error code

config: structure containing Nortek Aquadopp Profiler configuration details, parsed from the binary configuration data structures that are placed at the beginning of the raw binary files (as described earlier). For details about the configuration parameters, refer to the manufacturer documentation.

  • SN: instrument type and serial number
  • headSN: head serial number
  • boardFrequency: board frequency
  • headFrequency: head frequency
  • fwVersion: firmware version
  • hwRevision: hardware revision
  • PICversion: PIC code version
  • avgInterval: time the instrument is actively measuring within the profile interval
  • measurementInterval: time between each measurement (i.e., data output rate)
  • compassUpdateRate: rate at which compass readings are updated
  • nbeams: number of transducers/beams.
  • ncells: number of cells
  • npings: number of pings per profile
  • beamAngle: angle of beams
  • blankingDistance: distance from the sensor head to the start of the first measurement cell. See Nortek forum discussion for calculation details.
  • cellSize: size of each cell. See Nortek forum discussion for calculation details.
  • coordSys: coordinates system of raw data (BEAM, XYZ or ENU)
  • orientation: Up or Down indicates the direction transducers are facing
  • transformationMatrix: matrix, used to convert from beam to xyz coordinates
  • correlationThreshold: correlation threshold for resolving ambiguities
  • rawDataVelocityScaling: scale conversion from raw data to data in mat file - always converts all velocities to m/s, from either mm/s or 0.1 mm/s. (Nortek Software does this as well).

units: structure containing unit of measure for fields in structures above. For instance, units.pressure='decibar'.

Example: NortekADCP1504_20101022T000021Z.mat

NETCDF

NetCDF is a machine-independent data format offered by numerous institutions, particularly within the earth and ocean science communities. Additional resources are noted here.

Example: NORTEKAQDPRO2958_20110904T000042Z.nc

Discussion

To comment on this product, click Add Comment below.

  • No labels