Skip to end of metadata
Go to start of metadata

CODAR Quality Controlled Surface Currents

About CODAR

CODAR (Coastal Ocean Dynamics Applications Radar) is a technology that allows the measurement of surface ocean current velocities at a distance using the Doppler shift of reflected radio waves. The term “CODAR” is frequently used as a generic term, but it is strongly associated with CODAR Ocean Sensors, a California company that manufactures them. A more general term for CODAR is “HF (High-Frequency) Coastal Radar”.

CODAR works on the principle of Bragg scattering from ocean surface waves. Bragg scattering returns a signal to the transmitting antenna only when the ocean waves are moving either directly towards or directly away from the antenna.

Bragg scattering from ocean waves is also highly wavelength-specific; any given system is sensitive only to ocean waves with wavelengths one-half the wavelength of the radio waves it transmits. The Salish Sea / Strait of Georgia CODAR system (device details) used by ONC is a 25 MHz model. A radio frequency of 25 MHz corresponds to a radio wavelength of 12 metres, so the CODAR system is sensitive to ocean wavelengths of 6 metres. If ocean waves of this wavelength are entirely absent, the CODAR system is unable to measure surface currents. In practice, there is virtually always some wave energy in the 6-metre band, so this is generally not a problem. In extremely calm conditions, however, the data quality can be reduced because of the resulting lower signal-to-noise ratio.

The principle of Bragg scattering dictates that virtually all of the radio signal received by a CODAR antenna is being reflected back from ocean waves of a particular wavelength. The reflected signal is Doppler-shifted to higher or lower frequencies, depending on whether the ocean wave is moving, respectively, toward or away from the antenna. In the absence of ocean currents, this Doppler shift gives a measurement of the wave speed, just as a police radar gun measures the speed of a vehicle on the highway. If ocean currents are present, however, the Doppler shift gives a measurement of the wave speed PLUS the speed of the ocean current towards or away from the antenna. Because the speed of deep-water waves is known for a given wavelength, the wave speed can be subtracted from the total measured speed, resulting in a value for the speed of the ocean current alone.

The equations used in calculating ocean current velocities from the Bragg-scattering signal assume that the ocean waves being measured are “deep-water” waves--that is, that their wavelengths are less than twice the water depth. As depth decreases beyond this point, the waves will increasingly take on the character of “shallow-water” waves, and the assumptions used in calculating the ocean currents will become less and less valid. For a 25 MHz system, then, ocean currents measured in water depths of less than 3 metres should be viewed with suspicion. Note that the daily tidal range in the Strait of Georgia is on the order of 3 metres, so this effect will be time-varying for the current deployment.

CODAR Stations and Combiners

The individual CODAR stations are listed as individual devices. For example, the original two CODAR stations were at the Westshore Coal Terminal in Tsawwassen and at the Iona Wastewater Treatment Plant, near Vancouver Airport. The stations are displayed in the search tree under “Westshore Coal Terminal (VCOL)” and “Iona (VION)”, respectively. Each station measures the radial velocities of ocean surface currents; that is, towards and away from the station's antenna. The stations have been placed in locations such that their areas of coverage overlap considerably. In the overlapping region, it is possible to combine the radial data from the two stations into “total” velocities that may be resolved into their true north-south and east-west components. In this way, the stations have radial and raw data, while we list the combined data products on a separate, virtual, device, nominally found in the search tree as "Oceanographic Radar System" at the same level as the station search tree locations (for example: https://data.oceannetworks.ca/DataSearch?location=SOGCS&deviceCategory=OCEANOGRAPHICRADAR).

CODAR Data Products Overview

The CODAR stations and combiner virtual device have different data products available to them. The CODAR stations have manufacturer produced .RUV files (radial surface velocity data) and raw data formats: CODAR Raw RNG DataCODAR Raw CSS DataCODAR Raw RNG Data. The CODAR combiner virtual devices have manufacturer produced .TUV files (total surface current data). Both types of devices have manufacturer produced PNG and GIF plots of their respective radial and total surface velocities and, recently, we've added the ONC-developed CODAR Quality-Controlled Surface Currents data products. 

Oceans 2.0 API filterdataProductCode=CODARQCSC

The data formats and processing algorithms are described in detail below.

Revision History

  1. 20191001: Initial CODAR Quality Controlled Surface Currents data product released

Data Product Options

Formats

The CODAR quality controlled surface current data products are processed on-the-fly from the source RUV files (from CODAR stations with radial surface velocities). The formats are described below. All of these data products have filenames that end with _CLEAN.

PNG, PDF and GIF plots

PNG and PDF files contain a graphic representation of both station radial and combined total velocities, depending on what device/station the data product was requested fromStation files with radial currents end in the four letter CODAR site code (e.g. -VCOL_CLEAN), while the combined files (with total velocities) end in -TOTALS_CLEAN. The PDF variant bundles all the PNGs (up to a file size limit) in a single vector-graphics PDF with a filename corresponding to the extents of the PNGs. The GIF variant also bundles the PNG images into an animation with a 1 second frame rate and pause at the end of the animation. The GIF breaks into multiple files at UTC midnight unless the entire set spans less than 24 hours or has 6 or fewer files.

Oceans 2.0 API filter: extension=png,pdf,gif

An example radials plot:

An example totals plot:

An example animated totals GIF:

Note that the above examples are based on test data. Real data is usually better.

MAT and NetCDF processed data files

The processed, quality controlled data used to create the plots are available in these files. Note that the NetCDF product is not yet available. The MAT file structure is compatible with the HFRProgs structures, version 2.1.2. The modifications (detailed far below) and extensions on HFRProgs only add structures/arrays, while maintaining the original HFRProgs structure. Both Totals and Radials, the main structures, are indexed by time, each instance is a separate reading. ONC processing code carefully handles and aligns the Radials time stamps to create the Totals, processing one at time (it's not slow). Consideration is made for time-varying source files: .tuv and .grd files as  well as device attributes that change within or between deployments. The MAT file contains an accumulation of data; the file is broken at a size limit based on standard memory limits (~1 GB), similar to how the GIF file accumulates up to 1 day or the PDF accumulates up to ~100 MB of data.

Oceans 2.0 API filter: extension=mat,nc

MAT file data structures and definitions

This section describes the MAT file structure in detail. HFRProgs's README and code also documents these structures, so only relevent ones will be listed in detail.

Radials: a structure array containing radial surface current data. Each instance of the structure is a scan at a particular time. Details as follows:

  • U/V: surface current radial velocities as read from the .ruv files and subsequently tested and cleaned
  • LonLat: _x2 vector containing longitude and latitude pairs for each U/V value
  • Error: can be taken from the ETMP or STDV column from the RUV file
  • temporalQuality: is the ETMP column from the RUV file, so it's usually a duplicate of Error
  • spatialQuality: is the ESPC column from the RUV file
  • OtherMetadata: a structure containing additional metadata including the raw data and test parameters for the various radial tests.
  • QCflag: a structure containing the overallFlag QC flag and any other flag generated by the radial tests and cleaning. Flag values of 1 are a pass, 4 is a failure/bad. The flag vectors all match the size of the U/V vectors. We've removed the Radials.Flag vector is our code as QCflag supersedes it.
  • ProcessingSteps: a cell array of char vectors (strings) reporting on the various details of the processing that was done. Tip: to get a print out of the comments, try this: char(Radials(1).ProcessingSteps) 

Totals: a structure containing total surface data:

  • U/V: surface current total velocities calculated from the Radials structure
  • LonLat: _x2 vector containing longitude and latitude pairs for each U/V value


Meta: a structure array containing the following metadata fields:
  • deviceID: A unique identifier to represent the instrument within the Ocean Networks Canada data management and archiving system.
  • creationDate:Date and time (using ISO8601 format) that the data product was produced. This is a valuable indicator for comparing to other revisions of the same data product.
  • deviceName: A name given to the instrument.
  • deviceCode: A unique string for the instrument which is used to generate data product filenames.
  • deviceCategory: Device category to list under data search ('Echosounder').
  • deviceCategoryCode: Code representing the device category. Used for accessing webservices, as described here: API / webservice documentation (log in to see this link).
  • lat: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site latitude and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure..
  • lon: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site longitude and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • depth: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site depth and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • deviceHeading: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site heading and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • devicePitch: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site pitch and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • deviceRoll: Fixed value obtained at time of deployment. Will be NaN if mobile or if both site roll and device offset are null. If mobile, sensor information will be available in mobilePositionSensor structure.
  • siteName: Name corresponding to its latitude, longitude, depth position.
  • stationCode: Code representing the station or site. Used for accessing webservices, as described here: API / webservice documentation (log in to see this link).
  • locationName: The node of the Ocean Networks Canada observatory. Each location contains many sites.
  • dataQualityComments: In some cases, there are particular quality-related issues that are mentioned here. This is distinct from QAQC information contained in the data structure.
  • MobilePositionSensor.name: A cell array of sensor names for mobile position sensors. If not a mobile device, this will be an empty cell string.
  • MobilePositionSensor.sensorID: An array of unique identifiers of sensors that provide position data for mobile devices - this data may be used in this data product.
  • MobilePositionSensor.deviceID: An array of unique identifiers of sensors that provide position data for mobile devices - this data may be used in this data product.
  • MobilePositionSensor.dateFrom: An array of datenums denoting the range of applicability of each mobile position sensor - this data may be used in this data product.
  • MobilePositionSensor.dateTo: An array of datenums denoting the range of applicability of each mobile position sensor - this data may be used in this data product.
  • MobilePositionSensor.typeName: A cell array of sensor names for mobile position sensors. If not a mobile device, this will be an empty cell string. One of: Latitude, Longitude, Depth, COMPASS_SENSOR, Pitch, Roll.
  • MobilePositionSensor.offset: An array of offsets between the mobile position sensors' values and the position of the device (for instance, if cabled profiler has a depth sensor that is 1.2 m above the device, the offset will be -1.2m).
  • MobilePositionSensor.sensorTypeID: An array of unique identifiers for the sensor type.
  • MobilePositionSensor.correctedSensorID: An array of unique identifiers of sensors that provide corrected mobile positioning data. This is generally used for profiling deployments where the latency is corrected for: CTD casts primarily.
  • deploymentDateFrom: The date of the deployment on which the data was acquired.
  • deploymentDateTo: The date of the end of the deployment on which the data was acquired (may be omitted if still deployed).
  • samplingPeriod: Sampling rate of the instrument in seconds (maybe omitted on some devices that have no scalar sensors).
  • samplingPeriodDateFrom: matlab datenum of the start of the corresponding sample period (maybe omitted on some devices that have no scalar sensors).
  • samplingPeriodDateTo: matlab datenum of the end of the corresponding sample period (maybe omitted on some devices that have no scalar sensors).
  • searchID: unique number tracking this search request (not normally included).
  • Attribution: A structure array with Attribution information, ordered by importance and date. For internal users, go to the Network Console to configure the attributions. If an organization has more than one role it will be collated. If there are gaps in the date ranges, they are filled in with the default Ocean Networks Canada citation. If the "Attribution Required?" field is set to "No" on the Network Console then the citation will not appear. For data products with a attribution (except MAT files) and for users making products from a MAT file, if the special attribution is blank/null, then the company default attribution will be used and if it is also blank/null, then the final attribution will consist of the organization name and role: "Ocean Networks Canada (Owner, Collaborator)". Here are the fields:
    • acknowledgement: the acknowledgement text, note that if the special acknowledgement blank/null, the default acknowledgement is used.
    • startDate: datenum format
    • endDate: datenum format
    • organizationName
    • organizationRole

CODAR QARTOD Tests and Procedures

Flag values used, follow our QAQC flag scheme (more details here Quality Assurance Quality Control, although this page is intended primarily for scalar data), where, simply:

  • 0 - No test
  • 1 - Pass
  • 4 - Fail

QARTOD guidelines

From the QARTOD “Manual for Real-Time Quality Control of High Frequency Radar Surface Current Data” – Version 1 (May 2016). The summary table below presents the QARTOD QC tests for HF radar surface current data:

Test Type

Test Name

Status

Test Control

Signal (or Spectral) Processing

Signal-to-Noise Ratio (SNR) for Each Antenna (Test 1)

Required

Embedded

Cross Spectra Covariance Matrix Eigenvalues (Test 2)

Suggested

Embedded

Single and Dual Angle Solution - Direction of Arrival (DOA) Metrics (magnitude) (Test 3)

Suggested

Embedded

Single and Dual Angle Solution - Direction of Arrival (DOA) Function Widths (3 dB) (Test 4)

Suggested

Embedded

Positive Definiteness of 2×2 Signal Matrix (Test 5)

Suggested

Embedded

Radial Components

Syntax (Test 6)

Required

National

Max Threshold (Test 7)

Required

Local and National

Valid Location (Test 8)

Required

Local and National

Radial Count* (Test 9)

In development

Local and National

Spatial Median Filter (Test 10)

Suggested

Local and National

Temporal Gradient (Test 11)

Suggested

Local and National

Average Radial Bearing (Test 12)

Suggested

Local and National

Synthetic Radial (Test 13)

In development

Local and National

Total Vectors

Data Density Threshold (Test 14)

Required

Local and National

GDOP Threshold (Test 15)

Required

Local and National

Max Speed Threshold (Test 16)

Required

Local and National

Spatial Median Comparison (Test 17)

Required

Local and National

Further Details on Required Radial Velocity Tests (Section 3.4.2 Radial Tests of the QARTOD manual)

This set of tests is conducted during the development of the radial velocities, or upon the resultant radial velocities. These tests may be carried out at the local, regional and/or national network levels.

Test 6 – Syntax (Required) A collection of tests ensuring proper formatting and existence of fields within a radial file.

The radial file may be tested for proper parsing and content, for file format (hfrweralluv1.0, for example), site code, appropriate time stamp, site coordinates, antenna pattern type (measured or ideal, for DF systems), and internally consistent row/column specifications.

Flags

Condition

Codable Instructions

Fail = 4

One or more fields are corrupt or contain invalid data.

If “File Format” ≠ “hfrweralluv1.0”, flag = 4

Suspect = 3

N/A

N/A

Pass = 1

Applies for test pass condition.

N/A

Test Exception: None.

Test specifications to be established by operator. Acceptable files types, site codes, coordinates, APM names, etc., must be presented. For example, the national network performs the following suite of tests:

• All radial files acquired by HFRNet portals report the data timestamp in the filename. The filename timestamp must not be any more than 72 hours in the future relative to the portals’ system time.

• The file name timestamp must match the timestamp reported within the file.

• Radial data tables (Lon, Lat, U, V, ...) must not be empty.

• Radial data table columns stated must match the number of columns reported for each row (a useful test for catching partial or corrupted files).

• The site location must be within range: − 180 ≤ Longitude ≤ 180 − 90 ≤ Latitude ≤ 90.

• As a minimum, the following metadata must be defined:

o File type (LLUV)

o Site code

o Timestamp

o Site coordinates

o Antenna pattern type (measured or idealized)

o Time zone (only Coordinated Universal Time or Greenwich Mean Time accepted)

Test 7 - Max Threshold (Required) Ensures that a radial current speed is not unrealistically high.

The maximum radial speed threshold (RSPDMAX) represents the maximum reasonable surface radial velocity for the given domain.

Flags

Condition

Codable Instructions

Fail = 4

Radial current speed exceeds the maximum radial speed threshold.

If RSPD > RSPDMAX, flag = 4

Suspect = 3

N/A

N/A

Pass = 1

Radial current speed is less than or equal to the maximum radial speed threshold.

If RSPD ≤ RSPDMAX, flag = 1

Test Exception: None.

Test specifications to be established by operator. The maximum total speed threshold is 1 m/s for the West Coast of the United States and 3 m/s for the East/Gulf Coast domain. The threshold must vary by region. For example, the presence of the Gulf Stream dictates the higher threshold on the East Coast.

Test 8 – Valid Location (Required) Removes radial vectors placed over land or in other unmeasurable areas.

Radial vector coordinates are checked against a reference file containing information about which locations are over land or in an unmeasurable area (for example, behind an island or point of land). Radials in these areas will be flagged with a code (FLOC) in the radial file (+128 in CODAR radial files) and are not included in total vector calculations.

Flags

Condition

Codable Instructions

Fail = 4

Radial contains a user-defined location flag code in the radial file.

If FLOC exists, flag = 4

Suspect = 3

N/A

None

Pass = 1

Radial does not contain a user-defined location flag code in the radial file.

If FLOC does not exist, flag = 1

Test Exception: None.

Test specifications to be established by operator. For CODAR systems, the reference file is called AngSeg_XXXX.txt, where XXXX is the four-letter site code of the station and is located in the “RadialConfigs” folder. These vectors receive a code of +128 in the flag column of the radial text file. BF systems use pre-set grid locations for radials.

Further Details on Required Total Velocity Tests (section 3.4.3 Total Vectors of the QARTOD manual)

Test 14 - Data Density Threshold* (Required) Tests that a sufficient number of radial velocities exist to compute a total velocity vector.

A minimum number of radial velocities (RV_MIN) are required to construct a total velocity vector. RV_CNT is the number of radial velocities available to be used in the calculation.

Flags

Condition

Codable Instructions

Fail = 4

Insufficient number of radial velocities exist.

If RV_CNT < RV_MIN, flag = 4

Suspect = 3

N/A

N/A

Pass = 1

A sufficient number of radial velocities exist.

If RV_CNT ≥ RV_MIN, flag = 1

Test Exception: Does not apply to BF systems.

Test specifications to be established by operator. Recommend RV_MIN = 3

In CODAR software, this is set in line 1 of the AnalysisOptions.txt configuration file; the default value is 2.

Test 15 - GDOP Threshold (Required) Tests that the uncertainty in velocity due to the geometric relationship between radials is low enough for the vector to be considered valid.

GDOP (Geometric Dilution of Precision) is a scalar representing the contribution of the radial (bearing) geometry to uncertainty in velocity at a given gridpoint. Higher GDOP values indicate larger co-variances associated with the least square’s fit used in obtaining the solution. GDOP must be less than a maximum allowed value of GDOP_MAX to pass and less than a GDOP_HIGH value to not be considered suspect.

Flags

Condition

Codable Instructions

Fail = 4

Poor geometric relationship between radials yields a total vector with too much uncertainty to be valid.

If GDOP ≥ GDOP_MAX, flag = 4

Suspect = 3

The GDOP value associated with a total vector solution may be acceptable.

If GDOP < GDOP_MAX and GDOP ≥ GDOP_HIGH, flag = 3

Pass = 1

The GDOP associated with the total vector solution is sufficient.

If GDOP < GDOP_HIGH, flag = 1

Test Exception: None.

Test specifications to be established by operator.

The national network uses a GDOP_MAX of 10 and a more conservative value of 1.25 for near-real time applications such as Web display.

Test 16 - Max Speed Threshold (Required) Ensures that a total current speed is not unrealistically high.

Like the maximum radial velocity threshold, the maximum total speed threshold TSPDMAX represents the maximum reasonable surface velocity for the given domain.

Flags

Condition

Codable Instructions

Fail = 4

Total current speed exceeds the maximum total speed threshold.

If TSPD > TSPDMAX, flag = 4

Suspect = 3

N/A

None.

Pass = 1

Total current speed is below or equal to the maximum total speed threshold.

If TSPD ≤ TSPDMAX, flag = 1

Test Exception: None.

Test specifications to be established by operator.

The maximum total speed threshold is 1 m/s for the West Coast of the United States and 3 m/s for the East/Gulf Coast domain. The threshold must vary by region. For example, the presence of the Gulf Stream dictates the higher threshold on the East Coast.

ONC CODAR QARTOD Overall Implementation

This section describes the implementation of the QARTOD tests and procedures from the previous section to create the CODAR Quality Controlled Surface Current data products. The ONC implementation uses a custom extended version of an open source MATLAB® toolbox called HFR_PROGS (High Frequency Radar PROGramming Suite), found here: https://github.com/rowg/hfrprogs (slightly updated from the old 2.1.2 version found here: https://cencalarchive.org/~cocmpmb/COCMP-wiki/index.php/HFR_Progs_download_page. Our implementation of HFR_PROGS has significant extensions, which should be, at some future date, shared back to the github rowg group repository or our own. If not, contact us for the code. 

The workflow consists of reading SeaSonde .RUV (radial u & v) files containing hourly radial data for each CODAR station. The QARTOD tests prescribed in the previous section are applied to first to the radial data, as described below:

Radial data QA/QC workflow for each station:

  1. Read hourly .RUV file
  2. Apply Test 6 - Syntax Test: Verify the syntax in the header matches the expected metadata for each site.
    • File type = RUV
    • Site code
    • Timestamp
    • Site coordinates
    • Antenna pattern type (measured or idealized)
    • Time zone
  3. Apply Test 7 - Max Threshold Test:
    • Flag radial data exceeding RSPDMAX.
  4. Apply Test 8 - Valid Location Test:
    • Flag radial data over land (VFLG = 128).
  5. Spatial/Temporal Quality Test:
    • Flag data where ESPC or ETMP = 999.
  6. Output new radial RUV data including QA/QC test results.

Note that an additional test has been applied (Spatial/Temporal Quality Test) not defined in the QARTOD manual to flag radial data having a Spatial Quality (ESPC) or Temporal Quality (ETMP) flag of 999. These flags indicate non-calculable data points as determined by the SeaSonde software and are included as columns in the .RUV files.

The output from the radial data QA/QC workflow are modified hourly .RUV files for each CODAR site that include columns containing the QA/QC test results. After the hourly radial data from each CODAR station has undergone QA/QC, a new set of total vectors are generated (combined) using the ‘clean’ radial data. The workflow for generating total current vectors consists of inputting the ‘clean’ radial data, inputting raw total data files (.TUV) and total grid positions from the SeaSonde combiner computer, calculating a new grid of total vectors with HFR_PROGS/makeTotals, applying the prescribed QA/QC tests, then outputting a new set of total (.TUV) data.

Total data QA/QC workflow:

  1. Input ‘clean’ radial data from radial data QAQC workflow for each station and time stamp.
  2. Input .TUV total vector files from SeaSonde.
  3. Input combine_<LOCATION_CODE>.grd grid file from SeaSonde combiner computer for each network, where location code is VSOG for Strait of Georgia, VRUP for Prince Rupert, etc. The grid file applied is also selected from a time range of applicability
  4. Calculate new set of totals from clean radial data using HFR_PROGS. In this step, the spatial grid is provided from the combine_LOCN.grd file. Pre-determined temporal and spatial windows around each grid point are used to determine which radial vectors contribute to each total vector.
    • A spatial threshold (a search radius) is supplied to makeTotals for griding the radial data into total: SPATTHRES
    • A minimum number of contributing CODAR sites is required: MIN_NUM_SITES
  5. Apply Test 14 - Data Density Test:
    • Flag total vectors with RV_MIN < 3.
  6. Calculate GDOP values at each grid point using HFR_PROGS.
  7. Apply Test 15 - GDOP Threshold Test:
    • Flag total vectors with GDOP ≥ GDOP_MAX.
    • Optional GDOP tests are available but currently are not used. HFR_Progs produces four other error estimates in addition to the standard GDOP: GDOPMaxOrthog, GDOPRadErr, GDOPSites, FitDif. These are available in the MAT file structure if users wish to do additional filtering. ONC Data Specialists can also enable these in the live data products by changing the device attributes in Oceans 2.0 (current values are empty so that tests are not applied).
  8. Apply Test 16 - Max Speed Threshold:
    • Flag total vectors with TSPD > TSPD_MAX.
  9. Spatial Quality Test:
    • Flag data where UQUAL or VQUAL = 999.
  10. Output new total data (.TUV) including QA/QC test results.

Step 9 is an additional QA/QC test not prescribed by QARTOD that flags total data having a U or V standard deviation quality indicator (UQUAL/VQUAL) of 999 as indicated in the .TUV files produced by SeaSonde. UQUAL and VQUAL values of 999 indicate a non-calculable result for the grid point.

The outputs of these steps are available in the MAT, netCDF and other file products. In the MAT file product, the Radials structure is the cleaned RUV data, The raw RUV data is available as:

Radials(1).Uraw = Radials(1).OtherMetadata.RawData(3);
Radials(1).Vraw = Radials(1).OtherMetadata.RawData(4);


The Totals structure is the cleaned total surface velocities. The 'raw' totals U and V velocity components are available as Totals(1).Uraw  and Totals(1).Vraw, with raw meaning that these values are as makeTotals.m  has calculated in step 4, prior to step 5 thru 9 that test and clean the data, so raw in this case includes the testing and cleaning done in the calculation of the radial velocities. All of the processing is all done on demand and on-the-fly. We have found to it be quite quick. The bolded parameters above are tune-able via the Oceans 2.0 device attributes, and the values used stored in additional fields in the MAT file structure so that are easily available. The parameters used the various functions are in Totals.OtherMetadata.<function name>.parameters. The Totals.ProcessingSteps cell array contains summary information on the cleaning steps in particular. An additional structure exists within both the Radials and Totals that captures the flags generated by the tests, QCflag.

ONC CODAR Software Implementation

As noted above, these data products are generated by a modified version of HFRProgs suite foe Matlab, version 2.1.2. The products also utilize Rich Pawlowicz's M_MapA mapping package for MATLAB", version 1.4k, 2019, available online at www.eoas.ubc.ca/~rich/map.html. Many thanks to Rich for his great software. In the remainder of this section, the specific modified and extended functions from HFRProgs are detailed. The list below is not an exhaustive list of the full implementation, but is intended to give insight on the implementation.

File Inputs

Input

Description

Implementation

Raw radial files

.ruv files from Seasonde coming from each CODAR sites (antenna) and for the same time stamp

For each CODAR regions, contributing sites are properly identified

Master Grid file

The grid file that exists on the combiner computer.

These files are archived as data products and may made be available via data search. They are locaiton and time tagged so that the appropriate file is used for each time stamp of data

Raw total file

.tuv file produced by Seasonde for the same time stamp. This is needed for the Spatial Quality Test.

Load the file from the correct array location.

Device Attribute QC parameters

For the CODAR station devices and radial data products:

For the CODAR combiner virtual devices and the total data products:

Function Descriptions

getGridHeader – New function

The function getGridHeader was created to parse the header file using getRDLHeader as a model. It outputs the metadata names and values contained in the header. The creation date of the file can then easily be extracted using the HFRProgs function getNameValuePair, a function developed to select the elements from a cell array of names and values that matches a given string.

loadRDLFile – HFRProgs modified function

This function conveniently loads multiple radial files into one radial MATLAB structure.

Rorig = loadRDLFile(fnames, 'true');

The function loadRDLFile loads the core variables such as Longitude, Latitude, U, V, Velocity (renamed RadComp by loadRDLFile), Range, Bearing, Heading, Temporal Quality (renamed Error by loadRDLFile), and VectorFlag from the raw Seasonde radial file.

Some variables (e.g. ESPC - Spatial Quality) from the Seasonde radial file are not imported along with the other core variables in the radial MATLAB structure created by the loadRDLFile function. However, it is possible to keep the original raw data in the final MATLAB structure by setting the input variable ‘saverawdata’ as true (the default is false). The original raw data (‘RawData’) is then saved under the field ‘OtherMetadata’ in the radial MATLAB structure.

Note: values of Range, Bearing, and Heading in the radial MATLAB structure differs from the raw Seasonde radial file, since as stated in the loadRDLFile function a conversion is applied using the HFRProgs true2math function:

% Change direction to cartesian convention.  Note, however, that bearing

% will still point away from radar and heading will still point towards radar.

TOTAL.RangeBearHead(:,2:3) = true2math( TOTAL.RangeBearHead(:,2:3) );

The true2math function converts angles measured in degrees from North (North = 0 degrees) to angles measured in degrees from East (East = 0 degrees).

The ‘ProcessingSteps’ is a useful section of the radial MATLAB structure that keeps a record of the processing functions applied to the radial data in a cell array.

loadRDLFile2 is a basically the same function as loadRDLFile with few modifications:

  • loadRDLFile2 output Seasonde variables Spatial Quality (ESPC) and Temporal Quality (ETMP, with its original Seasonde name and instead of ‘Error’) in the radial MATLAB structure. RADIALmatvars was replaced by RADIALmatvars2 and includes variables:

RADIALmatvars2={'U','V','RadComp','ETMP','ESPC','QCflag','VectorFlag'};

  • The variable ‘Flag’ in loadRDLFile was renamed ‘QCflag’ in ‘QCflag’ contains a vector of flags 0 named ‘overallFlag’. It will be used to save the overall flag as QC tests are applied.
  • The lines of code in loadRDLFile replacing any 999 values from in the RADIALmatvars variables with NaN were commented out, in keeping with the function’s primary goal to load radial files into MATLAB. Any processing of the raw data will be taken care of by different functions in subsequent steps.

Note: As in loadRDLFile a vector Flag is created with default flag values of 1 and a processing step is added in the ‘ProcessingSteps’ cell array.


checkRDLSyntax – New function

This function was created to do the syntax checks required in the QARTOD manual for CODAR data. Specifically, checkRDLSyntax checks for:

  • File type is LLUV
  • Site code exists and is consistent with the site code reported in the file name
  • Time stamp exists
  • Time stamp matches time stamp reported in the file name
  • Core variable data exist
  • Number of columns matches number of column names
  • Site location exist
  • Site location is within proper range
  • Antenna pattern type
  • Time zone exists and is either UTC or GMT

The HFRProgs function getRDLHeader is used to parse the header of the radial files. The HFRProgs function getNameValuePair is used to extract the relevant information (e.g. the value ‘measured’ of the variable ‘AntennaPatternType’) from the parsed radial file header. Subsequent lines of code verify that the information exists and/or that the information is correct. If a condition is not respected an error is produced.

All the different syntax checks are saved in the ‘OtherMetadata’ section of the MATLAB structure under ‘cleanRDLSyntax’ field and a processing step is added in the ‘ProcessingSteps’ cell array.


cleanRadialsVel, cleanRadialsLoc, cleanRadialsTemp, cleanRadialsSpat – Modified and New functions

These functions were developed using the function cleanRadials from the HFRProgs as a modelFor each modified and new functions the list of variables affected by the result of the test was reduced to the following variables:

RADIALmatvars3 = {'U','V','RadComp'}

The RADIALmatvars3 list is directly defined inside each functions. Any data point failing a test values of the variables in RADIALmatvars3 are replaced by NaN and the corresponding value in the vector Flag is replaced by 4.

The HFRProgs function cleanRadials is a speed test. The function cleanRadialsVel is the essentially the same function as cleanRadials but it uses RADIALmatvars3 instead of RADIALmarvars. The threshold value, RSPDMAX, is an input variable as it is site specific and should be determined by ONC Science. From the ONC “Summary of Coastal Ocean Dynamics Application Radar (CODAR): Operation, Data Quality and Products”:

The CODAR manual recommends a maximum threshold of 1 m/s for the West Coast of the United States, and 3 m/s for the East/Gulf Coast; however, thresholds may vary by region. The RSPDMAX value used to test the radial velocity was 1.3 m/s for the Strait of Georgia CODAR data; however, the “best” value to be used operationally in real-time should be determined by considering observed and modeled surface current data for each operational region. Note that the SeaSonde software has a “Maximum Radial Velocity” threshold in the settings for determining the first order limits. This parameter limits the radial velocity magnitude in the output data by changing the Bragg spectral boundary limits used in processing, thus works in a different way than the QARTOD test.

cleanRadialsLoc is the function created to clean out the vector values 128 values from the ‘VectorFlag’ vector. The VectorFlag informs about the validity of location of the vector (on land, behind a structure such as an island, etc.). As described in the Seasonde ‘File_LLUV’ document (Sept 24, 2016):

Vector Flag

The Vector Indicator Flag (VFLG) in the fifth column is a composite value used by the Combine Grid and the LLUV files to indicate extra information about individual current vectors.

Bit0 (+1) disabled grid point and not outputted in LLUV files.

Bit1 (+2) grid point is near the coastline.

Bit2 (+4) grid point contains a point measurement like an ADCP.

Bit4 (+16) current vector was the result of interpolating across a baseline area.

Bit5 (+32) vector result exceeded the Maximum current limit.

Bit7 (+128) vector is out of bounds. For radial vectors this means that the vector was outside the Angular filter area. Not used for Total Vectors.

Bit8 (+256) Total vector does not have enough angular resolution from the contributing radials.

Bit9 (+512) vector is marked as hidden and should not be normally displayed.

Bit11(+2048) vector was created by interpolation.

Bit12(+4096) vector is of dubious quality and should not normally be used or displayed.

Currently the QARTOD manual only specifies that the vector flag VFLG = 128 need to be flagged as bad data and should not be included in the total vector calculation. The value 128 is the input variable FLOC in this function so it can easily be configured if QARTOD decides to include (for example) more or other VectorFlag values in the future.

cleanRadialsTemp and cleanRadialsSpat are the functions created to clean out the 999 values from, respectively, the Temporal Quality (ETMP) and the Spatial Quality (ESPT) vectors. As described in the Seasonde File_LLUV document (Sept 24, 2016):

Uncertainties/Quality Factors

There are three to six columns for Quality Factors. Their usage is dependent on the type of LLUV: radial, elliptical or total. While we typically call these Quality Factors, it's really a misnomer; they are really measurement uncertainties but it is simpler to label them as quality. They indicate measurement uncertainty due to noise, a changing surface current pattern, and/or horizontal shear over each measured area. They represent a lower bound on the actual uncertainty in the current velocity. For example, errors due to contaminated radar spectra or incorrect first-order region boundaries will not be included in the estimate. We believe that most spatial uncertainty is due to horizontal current shear and most temporal uncertainty is due to the current pattern changing over time. A resulting Quality Factor of 999 indicates a non-calculable result. For Radials & Ellipticals, there are 7 possible columns: Temporal (ETMP), Spatial (ESPC), Maximum Velocity (MAXV), Minimum Velocity (MINV), Velocity Doppler Count (EDVC), Spatial Count (ERSC), and Spatial Count (ERTC)

The 999 value is not likely to change and is hardcoded inside both cleanRadialsTemp and cleanRadialsSpat functions.

In the cleanRadialsTemp the function looks for either variables named ETMP or Error and can accept both.

The function cleanRadialsSpat was designed to look for the variable ESPT in the original raw data table saved in the ‘OtherMetadata’ section if loadRDLFile (with ‘saverawdata’ as true) was used to import the radial files instead of loadRDLFile2.

Each test thresholds are saved in the ‘OtherMetadata’ section of the radial MATLAB structure under the field with the respective function name and a processing step is added in the ‘ProcessingSteps’ cell array.

For each QC tests, a resulting vector of flags is saved under ‘QCflag’: a field ‘radialSpeedflag’, ‘locationFlag’, ‘temporalQualityFlag’, and ‘spatialQualityFlag’ are created after respectively, cleanRadialsVel, cleanRadialsLoc, cleanRadialsTemp and cleanRadialsSpat functions are used. For any data point failing one of these test, the corresponding value in the ‘overallFlag’ is changed to 4.


makeTotals - HFRProgs modified function

This function makeTotals uses the HFRProgs function tuvls that calculates the totals and different GDOP errors and output a total MATLAB structure.

[Totals, ~] = makeTotals(Radials, 'Grid', Grid.nodesOcean, 'TimeStamp', Radials(1).TimeStamp, 'spatthresh', spatialThreshold,...
'tempthresh', TEMPTHRESH, 'MinNumSites', minNumSites, 'MinNumRads', minNumRadials, ...
'WhichErrors', {'GDOPMaxOrthog', 'GDOP', 'GDOPRadErr', 'GDOPSites', 'FitDif'});

makeTotals also performs some density checks. There are two input variables relating to the density checks: minNumSites for minimum number of contributing sites, and MinNumRads for the minimum number of radials. If the number of sites is less than the threshold value a warning is issued. If the number of radials is less than the threshold value, calculation of totals from the given radials is skipped with the use of continue (no error or warning are generated).

The different GDOP calculated from the function makeTotals are: ’GDOPMaxOrthog', 'GDOPRadErr', 'GDOPSites', 'GDOP’, ‘FitDif'. For each GDOP type, Uerr, Verr, UVCovariance and TotalsErrors values are calculated and stored under the field ‘ErrorEstimates’ in the total MATLAB structure.

After contacting different members of the HF Radar community, Mark Otero from HFRNet group send us the piece of code National Network uses to calculate the GDOP. Following this, it was decided that ONC will calculate GDOP using all angles (named ‘GDOP’, in HFRProgs). Code used by the National Network to calculate the GDOP:

% UWLSTOTAL Computes a total velocity from radials using least-squares
%
%    [ u, v, dopx, dopy, hdop ] = uwlsTotal( rSpeed, rHeading ) returns the 
%    eastward (u) and northward (v) components of the total velocity 
%    resulting from the unweighted least squares fit of radial velocity 
%    measurements provided by rSpeed and rHeading.  The horizontal dilution
%    of precision (hdop) and corresponding eastward (dopx) and northward
%    (dopy) components are also returned as a metric for assessing the 
%    quality of the solution.
%
%    INPUT:
%        rSpeed   - Column vector (n* x 1) of radial velocity magnitude
%        rHeading - Column vector (n* x 1) of radial velocity heading in 
%        degrees counterclockwise from +x (east)
%
%        *n >= 2
%
%    OUTPUT:
%        u    - Eastward total velocity
%        v    - Northward total velocity
%        dopx - Eastward dilution of precision
%        dopy - Northward dilution of precision
%        hdop - Horizontal dilution of precision
%
%    All output values are scalars and the units of eastward and northward 
%   velocity are the same as the units for rSpeed.
%
%    Radial velocities are related to the total velocity by projection of 
%   the eastward and northward components of the total
%   velocity onto the radial heading:
%        rSpeed = u*cosd(rHeading) + v*sind(rHeading)
%
%    and in matrix form as
%        rSpeed = X*b
%
%    where
%        X = [ cosd(rHeading) sind(rHeading) ]
%        b = [ u; v ]
%
%    The unweighted least squares solution for b is given by
%        b = inv( X'*X ) * X' * rSpeed
%
%    The geometric portion of the covariance matrix for b is
%        C = inv( X'*X )
%
%    which provides the dilution of precision along the diagonal where
%        dopx = sqrt( C(1,1) )
%        dopy = sqrt( C(2,2) )
%        hdop = sqrt( C(1,1) + C(2,2) )
%
%    References
%    -   Least-squares methods for the extraction of surface currents from 
%    CODAR crossed-loop data: Application at ARSLOE, Lipa, B. J., D. E. 
%    Barrick, IEEE J. Ocean. Eng., 1983.
%    -  Shipborne measurement of surface current fields by HF radar, Gurgel,
%    K. -W., L'Onde Electr., 1994.
%    -   On the accuracy of HF radar suface current measurements: 
%    Intercomparisons with ship-based sensors, Chapman, R. D., L. K.  Shay,
%    H. C. Graber, J. B. Edson, A. Karachintsev, C. L. Trump, D.  B. Ross, 
%    J. Geophys. Res., 1997.
%    -  Guide to GPS Positioning, Wells, D. E., et al., Can. GPS Assoc., 
%    Fredericton, N. B., Canada, 1986.
%
%    Acknowledgement
%    This function was adapted from the TOTCALC function (ver. 3.0, Apr 
%    2004) included in the HFRadarmap toolbox (ver. 4.1, Sept 2004) written
%    by Mike Cook while at the Naval Postgraduate School Department of
%    Oceanography.
%
%    HF-Radar Network
%    Mark Otero
%    Scripps Institution of Oceanography
%    Coastal Observing Research and Development Center


makeTotals is basically the same function as the original makeTotals with the following modifications:

  • It was modified to accept the Temporal Quality ‘ETMP’ variable (line 180, 181 of the function)
  • The function will throw an error instead of a warning if the number of sites is less than the threshold (line 195).
  • The function creates a vector Flag of zeroes to be able to store flags after QC processing. (line 107)
  • The value in the vector Flag is replaced by 4 where the number of site is less than MINSITES or the number of radials is less than MINRAD (line 309 to 311).

Same as the radial QC functions, some information for ‘book keeping’ are saved. For this function some parameters (such as MINSITES and MINRAD thresholds, time stamp and other input parameters) are saved in the ‘OtherMetadata’ section in the total MATLAB structure, and a processing step with the name of the function is added to the ‘ProcessingSteps’ cell array.

A structure ‘QCFlag’ was created containing two fields: ‘overallFlag’ and ‘numSitesRadialsFlag’. ‘overallFlag’ is the vector containing the overall flag results after different QC tests are applied. The ‘numSitesRadialsFlag’ is the vector containing the results from the QC test based on the minimum number of sites and minimum number of radials contributing to the calculation of the total vector. Since this test is needed at (and integrated in) the makeTotals2 function level, the function output both ‘overallFlag’ and ‘numSitesRadialsFlag’ with both vectors being identical at that step.


cleanTotals - HFRProgs function, replaced by cleanTotalsVel - Modified function

The function cleanTotals is a speed test for totals. U and V values failing the test are replaced by NaN. The threshold value TSPDMAX is an input variable as it is site specific and should be determined by ONC Science. From the ONC “Summary of Coastal Ocean Dynamics Application Radar (CODAR): Operation, Data Quality and Products”:

The recommended threshold (TSPD_MAX) is 1 m/s for the West Coast of the United States and 3 m/s for the East and Gulf Coasts. The threshold will vary by region and appropriate values will be determined by operational region for ONC CODAR data.

The function cleanTotalsVel is the same function as cleanTotals except that for any given point failing the test the corresponding values in the vector ‘Velocity_Flag’ and in the ‘Overall_Flags’ vectors under ‘QCFlag’ are replaced by 4.

Relevant information (TSPDMAX,etc.) are saved in the ‘OtherMetadata’ section and a processing step with the name of the function is added to the ‘ProcessingSteps’ cell array in the total MATLAB structure.


cleanGDOP – New function

This function was created to calculate the GDOP value using the Uerr and Verr calculated by the makeTotals function, as follow:

sqrt(Uerr+Verr)

The threshold GDOPMAX is an input variable so it can easily be configured. For each GDOP values failing the test, corresponding U and V data points are replaced by NaN and the corresponding values in the ‘gdopFlag’ and in the ‘Overall_Flags’ vectors under ‘QCFlag’ are replaced by 4.

QARTOD recommends two values for the GDOPMAX. GDOPMAX = 10 is used by the National network and GDOPMAX = 1.25 is used for near real time application and web display. HFRNet uses 1.25.

GDOP type used in the cleanGDOP function and the threshold are saved in the ‘OtherMetadata’ section. The GDOP calculated error is saved in the ‘OtherMatrixVars’ section. A processing step with the name of the function is added to the ‘ProcessingSteps’ cell array in the total MATLAB structure.


cleanTUVQual – New function

This function was created to clean out the data points with corresponding UQAL and VQUAL equal to 999 in the total file (.tuv) produced by Seasonde. Prior using this function, the Seasonde total file needs to be loaded using the loadTUVFile, an HFRProgs function similar to loadRDLFile but for totals.

The function cleanTUVQual matches the data points by longitudes and latitudes between the Seasonde total file and the total MATLAB structure. U and V data points with corresponding UQAL and VQUAL values of 999 are replaced by NaN and the corresponding values in the ‘TUVQUAL_Flags’ and in the ‘Overall_Flags’ vectors under ‘QCFlag’ are replaced by 4.

A processing step with the name of the function is added to the ‘ProcessingSteps’ cell array in the total MATLAB structure, and an index vector and the UQAL and VQAL bad value (999) are saved in the ‘OtherMetadata’ section.


Data Availability

CODAR radial and total-current data is also available from HFRNet, a continent-wide repository of HF Radar measurements. An interactive mapping application there provides a graphical way to explore ocean currents, not only from Oceans 2.0, but from CODAR stations across the United States (and a single station in Mexico) as well.
Straight of Georgia Deployment

Radial data is available from the Westshore Coal Terminal station (VCOL) beginning on on November 24, 2011 and from the Iona Wastewater Treatment Plant (VION) beginning August 1, 2012. “Total” current data files are also available beginning August 1, 2012. Radial and total data files are generated hourly.

CSS and CSQ/Range data are available for each station for the same time period as the corresponding radial data. The exception to this is the CSQ files from the VCOL station in the five and half months between the date that acquisition started and June 10, 2012. An unnoticed default setting on the VCOL computer resulted in the CSQ files from this period being overwritten (apart from the CSQ files from February 6-7, 2012, which were recovered).

The VION station was switched from retaining CSQ files to retaining Range files on December 4, 2012. VCOL was switched to retaining Range files on December 19, 2012. Because VCOL is on a wireless router with limited bandwidth, CSQ/Range files are not fetched over the network, but rather are physically transferred via portable hard drive when site visits are made. As a result, there may be a lag of several months between acquisition and availability of CSQ/Range files from VCOL.

The range of high-frequency radio waves is greatly reduced over fresh water compared to the range over salty, highly-conductive water. For this reason, during the spring freshet, when the surface water in the Strait of Georgia is much less salty, one or both of the CODAR stations may return little or no current data in the area of overlap, resulting in no total current vectors being available. This will vary from year to year, but at the time of writing (May, 2013) it appears that total current vectors may be unavailable from early May to as late as July each year.

As of May 2016, the data acquisition framework was switched to the stations/combiners model. All of the formats will be available from then onward. The older files will migrate from the old scheme to the new devices over the next few months. The old scheme used a single device with multiple file mode fields to distinguish the stations. The new scheme is more flexible for expansion (we have added CODAR stations in Prince Rupert and there's potential for much more).

Discussion

To comment on this product, click Write a comment... below.

  • No labels