Oceans 3.0 API filter: dataProductCode=CODARQCSC
The data formats and processing algorithms are described in detail below.
The CODAR quality controlled surface current data products are processed on-the-fly from the source RUV files (from CODAR stations with radial surface velocities). The formats are described below. All of these data products have filenames that end with _CLEAN.
PNG and PDF files contain a graphic representation of both station radial and combined total velocities, depending on what device/station the data product was requested from. Station files with radial currents end in the four letter CODAR site code (e.g. -VCOL_CLEAN), while the combined files (with total velocities) end in -TOTALS_CLEAN. The PDF variant bundles all the PNGs (up to a file size limit) in a single vector-graphics PDF with a filename corresponding to the extents of the PNGs. The GIF variant also bundles the PNG images into an animation with a 1 second frame rate and pause at the end of the animation. The GIF breaks into multiple files at UTC midnight unless the entire set spans less than 24 hours or has 6 or fewer files.
Oceans 3.0 API filter: extension=png,pdf,gif
An example radials plot:
An example totals plot:
An example animated totals GIF:
Note that the above examples are based on test data. Real data is usually better.
The processed, quality controlled data used to create the plots are available in these files. Care has been taken to ensure the MAT file structure is compatible with the HFRProgs structures, version 2.1.2. The modifications (detailed far below) and extensions on HFRProgs only add structures/arrays, while maintaining the original HFRProgs structure. (One exception is the Flag sub-structure that was the most recent addition to HFRProgs and was empty, it has been replaced by the QC sub-structure.) Both Totals and Radials, the main structures, are indexed by time, each instance is a separate reading. ONC processing code carefully handles and aligns the Radials time stamps to create the Totals, processing one at time (it's not slow). Consideration is made for time-varying source files: .tuv and .grd files as well as device attributes that change within or between deployments. The MAT and netCDF files contain an accumulation of data; the file is broken at a size limit based on standard memory limits (~1 GB), similar to how the GIF file accumulates up to 1 day or the PDF accumulates up to ~100 MB of data. Radials structures may be excluded optionally. MAT files can contain both Totals and Radials structures, whereas the netCDF file cannot easily contain both, so separate files are created with -Radials and -Totals modifiers in the file-names. The structure of the netCDF file is closely related to the structure of the MAT file.
Oceans 3.0 API filter: extension=mat,nc
This section describes the MAT file structure in detail. HFRProgs's README and code also documents these structures, so only relevant ones will be listed in detail.
Radials: a structure array containing radial surface current data. Each instance of the structure is a scan at a particular time. Details as follows:
char(Radials(1).ProcessingSteps) 
Radials.QC: a structure containing the overallFlag QC flag and any other flag generated by the radial tests and cleaning. Flag values of 1 are a pass, 4 is a failure/bad. The flag vectors all match the size of the U/V vectors. We've removed the Radials.Flag vector in our code as the QC structure supersedes it. QC also contains all of the test parameters, named with the same test name as the flag appended by 'TestThreshold'. Unless noted in the documentation, the tests pass if the value is less than this threshold. All test thresholds are stored as device attributes and vary by device and time. A sub-structure contains the results of the .ruv file syntax test: CheckRDLsyntax, each value in contained within describes each check and it's result, all of which must pass for the data to be included in the Radials and Totals structures. Failures are noted in the search status (visible in the Data Search cart). Syntax failures should not happen.
Totals: a structure containing total surface data:
Totals.ErrorEstimates: struct array of the test results for each algorithm. One note-able difference here is that the GDOP test has a value in OtherMatrixVars. These are the new test values calculated as GDOPerror = sqrt(T.ErrorEstimates(i).Uerr + T.ErrorEstimates(i).Verr) while the test values are taken as the TotalErrors for all other tests.
Totals.OtherMetadata.makeTotals.parameters: this structure contains the parameter values used by makeTotals.m. This is unchanged from HFRProgs, other than all the 'WhichErrors' are used, the MinNumSites, MinNumRads, spatthresh, tempthresh are provided by the Oceans 3.0 device attributes, which are variable for each device and each time stamp.
Totals.QC: a structure containing the overallFlag QC flag and any other flag generated by the Totals tests and cleaning. Flag values of 1 are a pass, 4 is a failure/bad. The flag vectors all match the size of the U/V vectors. We've removed the Totals.Flag vector in our code as the QC structure supersedes it. QC also contains all of the test parameters, named with the same test name as the flag appended by 'TestThreshold'. Unless noted in the documentation, the tests pass if the value is less than this threshold. Note the test thresholds with values of 'Inf' are effectively not used, but are available if ONC Data Quality Specialists modify the device attributes with values to make use of these tests. All test thresholds are stored as device attributes and vary by device and time.
Grid: a structure containing all the information from accompanying .grid files primarily for CODAR combiner stations. It will also appear for CODAR radial stations where the .grid files are available as described in the CODAR grid files documentation.
NetCDF is a machine-independent data format offered by numerous institutions, particularly within the earth and ocean science communities. Additional resources are noted here. NetCDF is intended to be self-describing so only limited additional information is required here.
The CODAR QC netCDF data product largely mirrors the MAT file described in detail above. The differences include the separation of radial and total data into separate files (both may be provided in the default option for combiner stations), naming and metadata reporting. The totals files will have the lat/lon variables and x,y Easting/Northing dimensions as these values come from the CODAR combiner grid (see the Grid structure above). Note that the grid used is the full grid, including flagged locations (so the size of U/V variables is larger in the netCDF format compared to the MAT format or Manufacturer's TUV files). In the radials files, the grid information is only provided if available and is for use in gridding the data (radial data is not gridded). The totals files organize the data into grids with y,x and time dimensions, while the radials files use arrays for the current data, with dimensions of time and index, where index links the currents to their lat/lon sampling locations. The x,y dimensions are the grid centers relative to the grid origin. The totals files will also contain additional QC tests and quality indicators. The radials files from combiner stations will contain data from multiple radials stations; the platform_name variable will supply the station name / call sign (i.e. VION) for each time step. The global attributes will contain many useful fields, including processing_steps that will describe all of the steps and tests applied to the data.
|
|
Flag values used, follow our QAQC flag scheme (more details here Quality Assurance Quality Control, although this page is intended primarily for scalar data), where, simply:
From the QARTOD “Manual for Real-Time Quality Control of High Frequency Radar Surface Current Data” – Version 1 (May 2016). The summary table below presents the QARTOD QC tests for HF radar surface current data:
Test Type | Test Name | Status | Test Control |
Signal (or Spectral) Processing | Signal-to-Noise Ratio (SNR) for Each Antenna (Test 1) | Required | Embedded |
Cross Spectra Covariance Matrix Eigenvalues (Test 2) | Suggested | Embedded | |
Single and Dual Angle Solution - Direction of Arrival (DOA) Metrics (magnitude) (Test 3) | Suggested | Embedded | |
Single and Dual Angle Solution - Direction of Arrival (DOA) Function Widths (3 dB) (Test 4) | Suggested | Embedded | |
Positive Definiteness of 2×2 Signal Matrix (Test 5) | Suggested | Embedded | |
Radial Components | Syntax (Test 6) | Required | National |
Max Threshold (Test 7) | Required | Local and National | |
Valid Location (Test 8) | Required | Local and National | |
Radial Count* (Test 9) | In development | Local and National | |
Spatial Median Filter (Test 10) | Suggested | Local and National | |
Temporal Gradient (Test 11) | Suggested | Local and National | |
Average Radial Bearing (Test 12) | Suggested | Local and National | |
Synthetic Radial (Test 13) | In development | Local and National | |
Total Vectors | Data Density Threshold (Test 14) | Required | Local and National |
GDOP Threshold (Test 15) | Required | Local and National | |
Max Speed Threshold (Test 16) | Required | Local and National | |
Spatial Median Comparison (Test 17) | Required | Local and National |
This set of tests is conducted during the development of the radial velocities, or upon the resultant radial velocities. These tests may be carried out at the local, regional and/or national network levels.
Test 6 – Syntax (Required) A collection of tests ensuring proper formatting and existence of fields within a radial file. | ||
The radial file may be tested for proper parsing and content, for file format (hfrweralluv1.0, for example), site code, appropriate time stamp, site coordinates, antenna pattern type (measured or ideal, for DF systems), and internally consistent row/column specifications. | ||
Flags | Condition | Codable Instructions |
Fail = 4 | One or more fields are corrupt or contain invalid data. | If “File Format” ≠ “hfrweralluv1.0”, flag = 4 |
Suspect = 3 | N/A | N/A |
Pass = 1 | Applies for test pass condition. | N/A |
Test Exception: None. | ||
Test specifications to be established by operator. Acceptable files types, site codes, coordinates, APM names, etc., must be presented. For example, the national network performs the following suite of tests: • All radial files acquired by HFRNet portals report the data timestamp in the filename. The filename timestamp must not be any more than 72 hours in the future relative to the portals’ system time. • The file name timestamp must match the timestamp reported within the file. • Radial data tables (Lon, Lat, U, V, ...) must not be empty. • Radial data table columns stated must match the number of columns reported for each row (a useful test for catching partial or corrupted files). • The site location must be within range: − 180 ≤ Longitude ≤ 180 − 90 ≤ Latitude ≤ 90. • As a minimum, the following metadata must be defined: o File type (LLUV) o Site code o Timestamp o Site coordinates o Antenna pattern type (measured or idealized) o Time zone (only Coordinated Universal Time or Greenwich Mean Time accepted) | ||
Test 7 - Max Threshold (Required) Ensures that a radial current speed is not unrealistically high. | ||
The maximum radial speed threshold (RSPDMAX) represents the maximum reasonable surface radial velocity for the given domain. | ||
Flags | Condition | Codable Instructions |
Fail = 4 | Radial current speed exceeds the maximum radial speed threshold. | If RSPD > RSPDMAX, flag = 4 |
Suspect = 3 | N/A | N/A |
Pass = 1 | Radial current speed is less than or equal to the maximum radial speed threshold. | If RSPD ≤ RSPDMAX, flag = 1 |
Test Exception: None. | ||
Test specifications to be established by operator. The maximum total speed threshold is 1 m/s for the West Coast of the United States and 3 m/s for the East/Gulf Coast domain. The threshold must vary by region. For example, the presence of the Gulf Stream dictates the higher threshold on the East Coast. | ||
Test 8 – Valid Location (Required) Removes radial vectors placed over land or in other unmeasurable areas. | ||
Radial vector coordinates are checked against a reference file containing information about which locations are over land or in an unmeasurable area (for example, behind an island or point of land). Radials in these areas will be flagged with a code (FLOC) in the radial file (+128 in CODAR radial files) and are not included in total vector calculations. | ||
Flags | Condition | Codable Instructions |
Fail = 4 | Radial contains a user-defined location flag code in the radial file. | If FLOC exists, flag = 4 |
Suspect = 3 | N/A | None |
Pass = 1 | Radial does not contain a user-defined location flag code in the radial file. | If FLOC does not exist, flag = 1 |
Test Exception: None. | ||
Test specifications to be established by operator. For CODAR systems, the reference file is called AngSeg_XXXX.txt, where XXXX is the four-letter site code of the station and is located in the “RadialConfigs” folder. These vectors receive a code of +128 in the flag column of the radial text file. BF systems use pre-set grid locations for radials. | ||
Test 14 - Data Density Threshold* (Required) Tests that a sufficient number of radial velocities exist to compute a total velocity vector. | ||
A minimum number of radial velocities (RV_MIN) are required to construct a total velocity vector. RV_CNT is the number of radial velocities available to be used in the calculation. | ||
Flags | Condition | Codable Instructions |
Fail = 4 | Insufficient number of radial velocities exist. | If RV_CNT < RV_MIN, flag = 4 |
Suspect = 3 | N/A | N/A |
Pass = 1 | A sufficient number of radial velocities exist. | If RV_CNT ≥ RV_MIN, flag = 1 |
Test Exception: Does not apply to BF systems. | ||
Test specifications to be established by operator. Recommend RV_MIN = 3 In CODAR software, this is set in line 1 of the AnalysisOptions.txt configuration file; the default value is 2. | ||
Test 15 - GDOP Threshold (Required) Tests that the uncertainty in velocity due to the geometric relationship between radials is low enough for the vector to be considered valid. | ||
GDOP (Geometric Dilution of Precision) is a scalar representing the contribution of the radial (bearing) geometry to uncertainty in velocity at a given gridpoint. Higher GDOP values indicate larger co-variances associated with the least square’s fit used in obtaining the solution. GDOP must be less than a maximum allowed value of GDOP_MAX to pass and less than a GDOP_HIGH value to not be considered suspect. | ||
Flags | Condition | Codable Instructions |
Fail = 4 | Poor geometric relationship between radials yields a total vector with too much uncertainty to be valid. | If GDOP ≥ GDOP_MAX, flag = 4 |
Suspect = 3 | The GDOP value associated with a total vector solution may be acceptable. | If GDOP < GDOP_MAX and GDOP ≥ GDOP_HIGH, flag = 3 |
Pass = 1 | The GDOP associated with the total vector solution is sufficient. | If GDOP < GDOP_HIGH, flag = 1 |
Test Exception: None. | ||
Test specifications to be established by operator. The national network uses a GDOP_MAX of 10 and a more conservative value of 1.25 for near-real time applications such as Web display. | ||
Test 16 - Max Speed Threshold (Required) Ensures that a total current speed is not unrealistically high. | ||
Like the maximum radial velocity threshold, the maximum total speed threshold TSPDMAX represents the maximum reasonable surface velocity for the given domain. | ||
Flags | Condition | Codable Instructions |
Fail = 4 | Total current speed exceeds the maximum total speed threshold. | If TSPD > TSPDMAX, flag = 4 |
Suspect = 3 | N/A | None. |
Pass = 1 | Total current speed is below or equal to the maximum total speed threshold. | If TSPD ≤ TSPDMAX, flag = 1 |
Test Exception: None. | ||
Test specifications to be established by operator. The maximum total speed threshold is 1 m/s for the West Coast of the United States and 3 m/s for the East/Gulf Coast domain. The threshold must vary by region. For example, the presence of the Gulf Stream dictates the higher threshold on the East Coast. | ||
This section describes the implementation of the QARTOD tests and procedures from the previous section to create the CODAR Quality Controlled Surface Current data products. The ONC implementation uses a custom extended version of an open source MATLAB® toolbox called HFR_PROGS (High Frequency Radar PROGramming Suite), found here: https://github.com/rowg/hfrprogs (slightly updated from the old 2.1.2 version found here: https://cencalarchive.org/~cocmpmb/COCMP-wiki/index.php/HFR_Progs_download_page. Our implementation of HFR_PROGS has significant extensions, which should be, at some future date, shared back to the github rowg group repository or our own. If not, contact us for the code.
The workflow consists of reading SeaSonde .RUV (radial u & v) files containing hourly radial data for each CODAR station. The QARTOD tests prescribed in the previous section are applied to first to the radial data, as described below:
Radial data QA/QC workflow for each station:
Note that an additional test has been applied (Spatial/Temporal Quality Test) not defined in the QARTOD manual to flag radial data having a Spatial Quality (ESPC) or Temporal Quality (ETMP) flag of 999. These flags indicate non-calculable data points as determined by the SeaSonde software and are included as columns in the .RUV files.
The output from the radial data QA/QC workflow are modified hourly .RUV files for each CODAR site that include columns containing the QA/QC test results. After the hourly radial data from each CODAR station has undergone QA/QC, a new set of total vectors are generated (combined) using the ‘clean’ radial data. The workflow for generating total current vectors consists of inputting the ‘clean’ radial data, inputting raw total data files (.TUV) and total grid positions from the SeaSonde combiner computer, calculating a new grid of total vectors with HFR_PROGS/makeTotals, applying the prescribed QA/QC tests, then outputting a new set of total (.TUV) data.
Total data QA/QC workflow:
Step 9 is an additional QA/QC test not prescribed by QARTOD that flags total data having a U or V standard deviation quality indicator (UQUAL/VQUAL) of 999 as indicated in the .TUV files produced by SeaSonde. UQUAL and VQUAL values of 999 indicate a non-calculable result for the grid point.
The outputs of these steps are available in the MAT, netCDF and other file products. In the MAT file product, the Radials structure is the cleaned RUV data, The raw RUV data is available as:
Radials(1).Uraw = Radials(1).OtherMetadata.RawData(3); Radials(1).Vraw = Radials(1).OtherMetadata.RawData(4); |
The Totals structure is the cleaned total surface velocities. The 'raw' totals U and V velocity components are available as Totals(1).Uraw and Totals(1).Vraw, with raw meaning that these values are as makeTotals.m has calculated in step 4, prior to step 5 thru 9 that test and clean the data, so 'raw' in this case includes the testing and cleaning done in the calculation of the radial velocities. All of the processing is all done on demand and on-the-fly and the data processing is quite quick. The bolded parameters above are tune-able via the Oceans 3.0 device attributes, and the values used stored in additional fields in the MAT file structure so that are easily available. The parameters used in the various functions are in the QC structure and in Totals.OtherMetadata.makeTotals.parameters. The Totals.ProcessingSteps cell array contains summary information on the cleaning steps in particular. The QC sub-structure is an additional structure that exists within both the Radials and Totals that captures the flags and test thresholds generated by the tests.
As noted above, these data products are generated by a modified version of HFRProgs suite foe Matlab, version 2.1.2. The products also utilize Rich Pawlowicz's M_Map, A mapping package for MATLAB", version 1.4k, 2019, available online at www.eoas.ubc.ca/~rich/map.html. Many thanks to Rich for his great software. In the remainder of this section, the specific modified and extended functions from HFRProgs are detailed. The list below is not an exhaustive list of the full implementation, but is intended to give insight on the implementation.
Input | Description | Implementation |
Raw radial files | .ruv files from Seasonde coming from each CODAR sites (antenna) and for the same time stamp | For each CODAR regions, contributing sites are properly identified |
Master Grid file | The grid file that exists on the combiner computer. | These files are archived as data products and may made be available via data search. They are location and time tagged so that the appropriate file is used for each time stamp of data The combiner device is one-level up in the Oceans2.0 search-tree from each radial station so that the processing code can easily associate them. |
Raw total file | .tuv file produced by Seasonde for the same time stamp. This is needed for the Spatial Quality Test. | Load the file from the correct array location. |
For the CODAR station devices and radial data products:
For the CODAR combiner virtual devices and the total data products:
The function getGridHeader was created to parse the header file using getRDLHeader as a model. It outputs the metadata names and values contained in the header. The creation date of the file can then easily be extracted using the HFRProgs function getNameValuePair, a function developed to select the elements from a cell array of names and values that matches a given string.
This function conveniently loads multiple radial files into one radial MATLAB structure.
Rorig = loadRDLFile(fnames, 'true');
The function loadRDLFile loads the core variables such as Longitude, Latitude, U, V, Velocity (renamed RadComp by loadRDLFile), Range, Bearing, Heading, Temporal Quality (renamed Error by loadRDLFile), and VectorFlag from the raw Seasonde radial file.
Some variables (e.g. ESPC - Spatial Quality) from the Seasonde radial file are not imported along with the other core variables in the radial MATLAB structure created by the loadRDLFile function. However, it is possible to keep the original raw data in the final MATLAB structure by setting the input variable ‘saverawdata’ as true (the default is false). The original raw data (‘RawData’) is then saved under the field ‘OtherMetadata’ in the radial MATLAB structure.
Note: values of Range, Bearing, and Heading in the radial MATLAB structure differs from the raw Seasonde radial file, since as stated in the loadRDLFile function a conversion is applied using the HFRProgs true2math function:
|
The true2math function converts angles measured in degrees from North (North = 0 degrees) to angles measured in degrees from East (East = 0 degrees).
The ‘ProcessingSteps’ is a useful section of the radial MATLAB structure that keeps a record of the processing functions applied to the radial data in a cell array.
This function was modified to store Seasonde variables Spatial Quality (ESPC) and Temporal Quality (ETMP, with its original Seasonde name and instead of ‘Error’) in the Radials structure. The lines of code in loadRDLFile replacing any 999 values from in the RADIALmatvars variables with NaN were commented out, in keeping with the function’s primary goal to load radial files into MATLAB. Any processing of the raw data will be taken care of by different functions in subsequent steps.
This function was created to do the syntax checks required in the QARTOD manual for CODAR data. Specifically, checkRDLSyntax checks for:
The HFRProgs function getRDLHeader is used to parse the header of the radial files. The HFRProgs function getNameValuePair is used to extract the relevant information (e.g. the value ‘measured’ of the variable ‘AntennaPatternType’) from the parsed radial file header. Subsequent lines of code verify that the information exists and/or that the information is correct. If a condition is not respected an warning is written to the log and search result (visible in the Data Search cart) and the file is excluded.
All the different syntax checks are saved in the Radials.QC structure under ‘CleanRDLSyntax’ field and a processing step is added in the ‘ProcessingSteps’ cell array.
These functions were developed using the function cleanRadials from the HFRProgs as a model. For each modified and new functions the list of variables affected by the result of the test was reduced to the following variables:
RADIALmatvars = {'U','V','RadComp'}
The RADIALmatvars list is directly defined inside each functions. Any data point failing a test values of the variables in RADIALmatvars are replaced by NaN and the corresponding value in the vector Flag is replaced by 4.
The HFRProgs function cleanRadials is a speed test. The function cleanRadialsVel is the essentially the same function as cleanRadials but it uses an updated list of variables. The threshold value, RSPDMAX, is an input variable as it is site specific and should be determined by ONC Science. From the ONC “Summary of Coastal Ocean Dynamics Application Radar (CODAR): Operation, Data Quality and Products”:
… The CODAR manual recommends a maximum threshold of 1 m/s for the West Coast of the United States, and 3 m/s for the East/Gulf Coast; however, thresholds may vary by region. The RSPDMAX value used to test the radial velocity was 1.3 m/s for the Strait of Georgia CODAR data; however, the “best” value to be used operationally in real-time should be determined by considering observed and modeled surface current data for each operational region. Note that the SeaSonde software has a “Maximum Radial Velocity” threshold in the settings for determining the first order limits. This parameter limits the radial velocity magnitude in the output data by changing the Bragg spectral boundary limits used in processing, thus works in a different way than the QARTOD test. … |
cleanRadialsLoc is the function created to clean out the vector values 128 values from the ‘VectorFlag’ vector. The VectorFlag informs about the validity of location of the vector (on land, behind a structure such as an island, etc.). As described in the Seasonde ‘File_LLUV’ document (Sept 24, 2016):
… Vector Flag The Vector Indicator Flag (VFLG) in the fifth column is a composite value used by the Combine Grid and the LLUV files to indicate extra information about individual current vectors. Bit0 (+1) disabled grid point and not outputted in LLUV files. Bit1 (+2) grid point is near the coastline. Bit2 (+4) grid point contains a point measurement like an ADCP. Bit4 (+16) current vector was the result of interpolating across a baseline area. Bit5 (+32) vector result exceeded the Maximum current limit. Bit7 (+128) vector is out of bounds. For radial vectors this means that the vector was outside the Angular filter area. Not used for Total Vectors. Bit8 (+256) Total vector does not have enough angular resolution from the contributing radials. Bit9 (+512) vector is marked as hidden and should not be normally displayed. Bit11(+2048) vector was created by interpolation. Bit12(+4096) vector is of dubious quality and should not normally be used or displayed. … |
Currently the QARTOD manual only specifies that the vector flag VFLG = 128 need to be flagged as bad data and should not be included in the total vector calculation. The value 128 is the input variable FLOC in this function so it can easily be configured if QARTOD decides to include (for example) more or other VectorFlag values in the future.
cleanRadialsTemp and cleanRadialsSpat are the functions created to clean out the 999 values from, respectively, the Temporal Quality (ETMP) and the Spatial Quality (ESPT) vectors. As described in the Seasonde File_LLUV document (Sept 24, 2016):
… Uncertainties/Quality Factors There are three to six columns for Quality Factors. Their usage is dependent on the type of LLUV: radial, elliptical or total. While we typically call these Quality Factors, it's really a misnomer; they are really measurement uncertainties but it is simpler to label them as quality. They indicate measurement uncertainty due to noise, a changing surface current pattern, and/or horizontal shear over each measured area. They represent a lower bound on the actual uncertainty in the current velocity. For example, errors due to contaminated radar spectra or incorrect first-order region boundaries will not be included in the estimate. We believe that most spatial uncertainty is due to horizontal current shear and most temporal uncertainty is due to the current pattern changing over time. A resulting Quality Factor of 999 indicates a non-calculable result. For Radials & Ellipticals, there are 7 possible columns: Temporal (ETMP), Spatial (ESPC), Maximum Velocity (MAXV), Minimum Velocity (MINV), Velocity Doppler Count (EDVC), Spatial Count (ERSC), and Spatial Count (ERTC) … |
The 999 value is not likely to change and is hardcoded inside both cleanRadialsTemp and cleanRadialsSpat functions.
cleanRadialsTemp the function works on the temporalQuality vector (ERROR or ETMP in the ruv header), while cleanRadialsSpat works on the spatialQuality vector (ESPT in the ruv header).
Each test threshold issaved in the QC structure with the respective name append by 'TestThreshold' and a processing step is added in the ‘ProcessingSteps’ cell array.
For each QC tess, a resulting vector of flags is saved under ‘QC’: a field ‘radialSpeedflag’, ‘locationFlag’, ‘temporalQualityFlag’, and ‘spatialQualityFlag’ are created after respectively, cleanRadialsVel, cleanRadialsLoc, cleanRadialsTemp and cleanRadialsSpat functions are used. For any data point failing one of these test, the corresponding value in the ‘overallFlag’ is changed to 4.
This function makeTotals uses the HFRProgs function tuvls that calculates the totals and different GDOP errors and output a total MATLAB structure.
[Totals, ~] = makeTotals(Radials, 'Grid', Grid.nodesOcean, 'TimeStamp', Radials(1).TimeStamp, 'spatthresh', spatialThreshold,...'tempthresh', TEMPTHRESH, 'MinNumSites', minNumSites, 'MinNumRads', minNumRadials, ...'WhichErrors', {'GDOPMaxOrthog', 'GDOP', 'GDOPRadErr', 'GDOPSites', 'FitDif'});
makeTotals also performs some density checks. There are two input variables relating to the density checks: minNumSites for minimum number of contributing sites, and MinNumRads for the minimum number of radials. If the number of sites is less than the threshold value a warning is issued. If the number of radials is less than the threshold value, calculation of totals from the given radials is skipped with the use of continue (no error or warning are generated).
The different GDOP calculated from the function makeTotals are: ’GDOPMaxOrthog', 'GDOPRadErr', 'GDOPSites', 'GDOP’, ‘FitDif'. For each GDOP type, Uerr, Verr, UVCovariance and TotalsErrors values are calculated and stored under the field ‘ErrorEstimates’ in the total MATLAB structure.
After contacting different members of the HF Radar community, Mark Otero from HFRNet group send us the piece of code National Network uses to calculate the GDOP. Following this, it was decided that ONC will calculate GDOP using all angles (named ‘GDOP’, in HFRProgs). Code used by the National Network to calculate the GDOP:
% UWLSTOTAL Computes a total velocity from radials using least-squares % % [ u, v, dopx, dopy, hdop ] = uwlsTotal( rSpeed, rHeading ) returns the % eastward (u) and northward (v) components of the total velocity % resulting from the unweighted least squares fit of radial velocity % measurements provided by rSpeed and rHeading. The horizontal dilution % of precision (hdop) and corresponding eastward (dopx) and northward % (dopy) components are also returned as a metric for assessing the % quality of the solution. % % INPUT: % rSpeed - Column vector (n* x 1) of radial velocity magnitude % rHeading - Column vector (n* x 1) of radial velocity heading in % degrees counterclockwise from +x (east) % % *n >= 2 % % OUTPUT: % u - Eastward total velocity % v - Northward total velocity % dopx - Eastward dilution of precision % dopy - Northward dilution of precision % hdop - Horizontal dilution of precision % % All output values are scalars and the units of eastward and northward % velocity are the same as the units for rSpeed. % % Radial velocities are related to the total velocity by projection of % the eastward and northward components of the total % velocity onto the radial heading: % rSpeed = u*cosd(rHeading) + v*sind(rHeading) % % and in matrix form as % rSpeed = X*b % % where % X = [ cosd(rHeading) sind(rHeading) ] % b = [ u; v ] % % The unweighted least squares solution for b is given by % b = inv( X'*X ) * X' * rSpeed % % The geometric portion of the covariance matrix for b is % C = inv( X'*X ) % % which provides the dilution of precision along the diagonal where % dopx = sqrt( C(1,1) ) % dopy = sqrt( C(2,2) ) % hdop = sqrt( C(1,1) + C(2,2) ) % % References % - Least-squares methods for the extraction of surface currents from % CODAR crossed-loop data: Application at ARSLOE, Lipa, B. J., D. E. % Barrick, IEEE J. Ocean. Eng., 1983. % - Shipborne measurement of surface current fields by HF radar, Gurgel, % K. -W., L'Onde Electr., 1994. % - On the accuracy of HF radar suface current measurements: % Intercomparisons with ship-based sensors, Chapman, R. D., L. K. Shay, % H. C. Graber, J. B. Edson, A. Karachintsev, C. L. Trump, D. B. Ross, % J. Geophys. Res., 1997. % - Guide to GPS Positioning, Wells, D. E., et al., Can. GPS Assoc., % Fredericton, N. B., Canada, 1986. % % Acknowledgement % This function was adapted from the TOTCALC function (ver. 3.0, Apr % 2004) included in the HFRadarmap toolbox (ver. 4.1, Sept 2004) written % by Mike Cook while at the Naval Postgraduate School Department of % Oceanography. % % HF-Radar Network % Mark Otero % Scripps Institution of Oceanography % Coastal Observing Research and Development Center |
makeTotals is basically the same function as the original makeTotals with the following modifications:
Same as the radial QC functions, some information for ‘book keeping’ are saved. For this function some parameters (such as MINSITES and MINRAD thresholds, time stamp and other input parameters) are saved in the ‘OtherMetadata’ section in the total MATLAB structure, and a processing step with the name of the function is added to the ‘ProcessingSteps’ cell array.
A structure ‘QC’ was created containing two fields: ‘overallFlag’ and ‘numSitesRadialsFlag’. ‘overallFlag’ is the vector containing the overall flag results after different QC tests are applied. The ‘numSitesRadialsFlag’ is the vector containing the results from the QC test based on the minimum number of sites and minimum number of radials contributing to the calculation of the total vector. Since this test is needed at (and integrated in) the makeTotals function level, the function output both ‘overallFlag’ and ‘numSitesRadialsFlag’ with both vectors being identical at that step.
The function cleanTotals is a speed test for totals. U and V values failing the test are replaced by NaN. The threshold value TSPDMAX is an input variable as it is site specific and should be determined by ONC Science. From the ONC “Summary of Coastal Ocean Dynamics Application Radar (CODAR): Operation, Data Quality and Products”:
… The recommended threshold (TSPD_MAX) is 1 m/s for the West Coast of the United States and 3 m/s for the East and Gulf Coasts. The threshold will vary by region and appropriate values will be determined by operational region for ONC CODAR data. … |
The function cleanTotalsVel is the same function as cleanTotals except that for any given point failing the test the corresponding values in the vector ‘totalSpeedFlag’ and in the ‘overallFlag’ vectors under ‘QC’ are replaced by 4. The max speed threshold is Totals.QC.totalSpeedMaxTestThreshold
Relevant information (TSPDMAX,etc.) are saved in the ‘OtherMetadata’ section and a processing step with the name of the function is added to the ‘ProcessingSteps’ cell array in the total MATLAB structure.
This function was created to calculate the GDOP value using the Uerr and Verr calculated by the makeTotals function, as follow:
sqrt(Uerr+Verr)
The threshold GDOPMAX is an input variable so it can easily be configured. For each GDOP values failing the test, corresponding U and V data points are replaced by NaN and the corresponding values in the ‘gdopFlag’ and in the ‘overallFlag’ vectors under ‘QC’ are replaced by 4.
QARTOD recommends two values for the GDOPMAX. GDOPMAX = 10 is used by the National network and GDOPMAX = 1.25 is used for near real time application and web display. HFRNet uses 1.25.
The GDOP calculated error is saved in the ‘OtherMatrixVars’ section. A processing step with the name of the function is added to the ‘ProcessingSteps’ cell array in the total MATLAB structure.
This function also implements the other GDOP tests, as specified by device attributes: 'GDOPMaxOrthogErrorMax', 'GDOPRadErrErrorMax', 'GDOPSitesErrorMax', 'FitDifErrorMax'. Currently, these are not used (default values in device attributes are null, values in the MAT file structure are set to Inf). The test metric is the TotalError value in the struct array Totals.ErrorEstimates and the test is simply if the test metric exceeds the threshold, the test fails, surface currents are NaN'ed and the corresponding QC flag is set to 4.
This function was created to clean out the data points with corresponding UQAL and VQUAL equal to 999 in the total file (.tuv) produced by Seasonde. Prior using this function, the Seasonde total file needs to be loaded using the loadTUVFile, an HFRProgs function similar to loadRDLFile but for totals.
The function cleanTUVQual matches the data points by longitudes and latitudes between the Seasonde total file and the total MATLAB structure. U and V data points with corresponding UQAL and VQUAL values of 999 are replaced by NaN and the corresponding values in the ‘tuvQualityFlag’ and in the ‘overallFlag’ vectors under ‘QC’ are replaced by 4.
A processing step with the name of the function is added to the ‘ProcessingSteps’ cell array in the total MATLAB structure..
To comment on this product, click Write a comment... below.