...
Oceans 2.0 API filter: extension=png,pdf,gif
An example radials plot:
An example totals plot:
...
The processed, quality controlled data used to create the plots are available in these files. Note that the NetCDF product is not yet available. The Care has been taken to ensure the MAT file structure is compatible with the HFRProgs structures, version 2.1.2. The modifications (detailed far below) and extensions on HFRProgs only add structures/arrays, while maintaining the original HFRProgs structure. (One exception is the Flag sub-structure that was the most recent addition to HFRProgs and was empty, it has been replaced by the QC sub-structure.) Both Totals and Radials, the main structures, are indexed by time, each instance is a separate reading. ONC processing code carefully handles and aligns the Radials time stamps to create the Totals, processing one at time (it's not slow). Consideration is made for time-varying source files: .tuv and .grd files as well as device attributes that change within or between deployments. The MAT file contains an accumulation of data; the file is broken at a size limit based on standard memory limits (~1 GB), similar to how the GIF file accumulates up to 1 day or the PDF accumulates up to ~100 MB of data.
...
This section describes the MAT file structure in detail. HFRProgs's README and code also documents these structures, so only relevent ones relevant ones will be listed in detail.
...
- U/V: surface current radial velocities as read from the .ruv files and subsequently tested and cleaned
- LonLat: _x2 :,2 vector containing longitude and latitude pairs for each U/V value
- Error: can be taken from the ETMP or STDV column from the RUV file (may be deprecated in the future)
- temporalQuality: is the ETMP column from the RUV file, so it's usually a duplicate of Error
- spatialQuality: is the ESPC column from the RUV file
- OtherMetadata: a structure containing additional metadata including the raw data and test parameters for the various radial tests.QCflag: a structure containing the this structure was modified from HFRProgs. We moved the ruv file Header and RawData into the Radials structure, see the next two entries:
- ruvHeader: cell strings of the ruv file header
- ruvRawData: matrix corresonding to the raw data body of the ruv file
- ProcessingSteps: a cell array of char vectors (strings) reporting on the various details of the processing that was done. Tip: to get a print out of the comments, try this:
char(Radials(1).ProcessingSteps)
Radials.QC: a structure containing the overallFlag QC flag and any other flag generated by the radial tests and cleaning. Flag values of 1 are a pass, 4 is a failure/bad. The flag vectors all match the size of the U/V vectors. We've removed the Radials.Flag vector is our code as
...
the QC structure supersedes it.
...
QC also contains all of the test parameters, named with the same test name as the flag appended by 'TestThreshold'. Unless noted in the documentation, the tests pass if the value is less than this threshold. All test thresholds are stored as device attributes and vary by device and time. A sub-structure contains the results of the .ruv file syntax test: CheckRDLsyntax, each value in contained within describes each check and it's result, all of which must pass for the data to be included in the Radials and Totals structures. Failures are noted in the search status (visible in the Data Search cart). Syntax failures should not happen.
Totals: a structure containing total surface data:
- U/V: surface current total velocities calculated from the Radials structure
- LonLat: _x2 vector containing longitude and latitude pairs for each U/V value
- U/V: surface current total velocities calculated from the Radials structure
- LonLat: :,2 vector containing longitude and latitude pairs for each U/V value
- Uraw/Vraw: surface current total velocities as generated by makeTotals.m prior to cleaning with subsequent tests on the the Totals, but these values were calculated from clean Radials and so are not strictly 'raw' data.
Totals.ErrorEstimates: struct array of the test results for each algorithm. One note-able difference here is that the GDOP test has a value in OtherMatrixVars. These are the new test values calculated as GDOPerror = sqrt(T.ErrorEstimates(i).Uerr + T.ErrorEstimates(i).Verr) while the test values are taken as the TotalErrors for all other tests.
Totals.OtherMetadata.makeTotals.parameters: this structure contains the parameter values used by makeTotals.m. This is unchanged from HFRProgs, other than all the 'WhichErrors' are used, the MinNumSites, MinNumRads, spatthresh, tempthresh are provided by the Oceans 2.0 device attributes, which are variable for each device and each time stamp.
Totals.QC: a structure containing the overallFlag QC flag and any other flag generated by the Totals tests and cleaning. Flag values of 1 are a pass, 4 is a failure/bad. The flag vectors all match the size of the U/V vectors. We've removed the Totals.Flag vector is our code as the QC structure supersedes it. QC also contains all of the test parameters, named with the same test name as the flag appended by 'TestThreshold'. Unless noted in the documentation, the tests pass if the value is less than this threshold. Note the test thresholds with values of 'Inf' are effectively not used, but are available if ONC Data Quality Specialists modify the device attributes with values to make use of these tests. All test thresholds are stored as device attributes and vary by device and time.
Include Page MAT file Metadata structure - Complex MAT file Metadata structure - Complex
...
The Totals structure is the cleaned total surface velocities. The 'raw' totals U and V velocity components are available as Totals(1).Uraw and Totals(1).Vraw, with raw meaning that these values are as makeTotals.m has calculated in step 4, prior to step 5 thru 9 that test and clean the data, so 'raw' in this case includes the testing and cleaning done in the calculation of the radial velocities. All of the processing is all done on demand and on-the-fly . We have found to it be and the data processing is quite quick. The bolded parameters above are tune-able via the Oceans 2.0 device attributes, and the values used stored in additional fields in the MAT file structure so that are easily available. The parameters used in the various functions are in the QC structure and in Totals.OtherMetadata.<function name>makeTotals.parameters. The Totals.ProcessingSteps cell array contains summary information on the cleaning steps in particular. An The QC sub-structure is an additional structure that exists within both the Radials and Totals that captures the flags and test thresholds generated by the tests, QCflag.
ONC CODAR Software Implementation
...
The ‘ProcessingSteps’ is a useful section of the radial MATLAB structure that keeps a record of the processing functions applied to the radial data in a cell array.
loadRDLFile2 is a basically the same function as loadRDLFile with few modifications:
...
This function was modified to store Seasonde variables Spatial Quality (ESPC) and Temporal Quality (ETMP, with its original Seasonde name and
...
RADIALmatvars2={'U','V','RadComp','ETMP','ESPC','QCflag','VectorFlag'};
...
instead of ‘Error’) in the Radials structure. The lines of code in loadRDLFile replacing any 999 values from in the RADIALmatvars variables with NaN were commented out, in keeping with the function’s primary goal to load radial files into MATLAB. Any processing of the raw data will be taken care of by different functions in subsequent steps.
Note: As in loadRDLFile a vector Flag is created with default flag values of 1 and a processing step is added in the ‘ProcessingSteps’ cell array.
checkRDLSyntax – New function
...
- File type is LLUV
- Site code exists and is consistent with the site code reported in the file name
- Time stamp exists
- Time stamp matches time stamp reported in the file name
- Core variable data exist
- Number of columns matches number of column names
- Site location exist
- Site location is within proper range
- Antenna pattern type
- Time zone exists and is either UTC or GMT
The HFRProgs function getRDLHeader is used to parse the header of the radial files. The HFRProgs function getNameValuePair is used to extract the relevant information (e.g. the value ‘measured’ of the variable ‘AntennaPatternType’) from the parsed radial file header. Subsequent lines of code verify that the information exists and/or that the information is correct. If a condition is not respected an error is produced.warning is written to the log and search result (visible in the Data Search cart) and the file is excluded.
All the different syntax checks are saved in the ‘OtherMetadata’ section of the MATLAB Radials.QC structure under ‘cleanRDLSyntax’ ‘CleanRDLSyntax’ field and a processing step is added in the ‘ProcessingSteps’ cell array.
...
These functions were developed using the function cleanRadials from the HFRProgs as a model. For each modified and new functions the list of variables affected by the result of the test was reduced to the following variables:
RADIALmatvars3 RADIALmatvars = {'U','V','RadComp'}
The RADIALmatvars3 RADIALmatvars list is directly defined inside each functions. Any data point failing a test values of the variables in RADIALmatvars3 RADIALmatvars are replaced by NaN and the corresponding value in the vector Flag is replaced by 4.
The HFRProgs function cleanRadials is a speed test. The function cleanRadialsVel is the essentially the same function as cleanRadials but it uses RADIALmatvars3 instead of RADIALmarvarsan updated list of variables. The threshold value, RSPDMAX, is an input variable as it is site specific and should be determined by ONC Science. From the ONC “Summary of Coastal Ocean Dynamics Application Radar (CODAR): Operation, Data Quality and Products”:
...
The 999 value is not likely to change and is hardcoded inside both cleanRadialsTemp and cleanRadialsSpat functions.
In the cleanRadialsTemp the function looks for either variables named ETMP or Error and can accept both.
The function cleanRadialsSpat was designed to look for the variable ESPT in the original raw data table saved in the ‘OtherMetadata’ section if loadRDLFile (with ‘saverawdata’ as true) was used to import the radial files instead of loadRDLFile2.
works on the temporalQuality vector (ERROR or ETMP in the ruv header), while cleanRadialsSpat works on the spatialQuality vector (ESPT in the ruv header).
Each test threshold issaved in the QC structure with the respective name append by 'TestThreshold' Each test thresholds are saved in the ‘OtherMetadata’ section of the radial MATLAB structure under the field with the respective function name and a processing step is added in the ‘ProcessingSteps’ cell array.
For each QC teststess, a resulting vector of flags is saved under ‘QCflag’‘QC’: a field ‘radialSpeedflag’, ‘locationFlag’, ‘temporalQualityFlag’, and ‘spatialQualityFlag’ are created after respectively, cleanRadialsVel, cleanRadialsLoc, cleanRadialsTemp and cleanRadialsSpat functions are used. For any data point failing one of these test, the corresponding value in the ‘overallFlag’ is changed to 4.
...
Same as the radial QC functions, some information for ‘book keeping’ are saved. For this function some parameters (such as MINSITES and MINRAD thresholds, time stamp and other input parameters) are saved in the ‘OtherMetadata’ section in the total MATLAB structure, and a processing step with the name of the function is added to the ‘ProcessingSteps’ cell array.
A structure ‘QCFlag’ ‘QC’ was created containing two fields: ‘overallFlag’ and ‘numSitesRadialsFlag’. ‘overallFlag’ is the vector containing the overall flag results after different QC tests are applied. The ‘numSitesRadialsFlag’ is the vector containing the results from the QC test based on the minimum number of sites and minimum number of radials contributing to the calculation of the total vector. Since this test is needed at (and integrated in) the makeTotals2 function makeTotals function level, the function output both ‘overallFlag’ and ‘numSitesRadialsFlag’ with both vectors being identical at that step.
...
The function cleanTotalsVel is the same function as cleanTotals except that for any given point failing the test the corresponding values in the vector ‘Velocity_Flag’ ‘totalSpeedFlag’ and in the ‘Overall_Flags’ ‘overallFlag’ vectors under ‘QCFlag’ ‘QC’ are replaced by 4. The max speed threshold is Totals.QC.totalSpeedMaxTestThreshold
Relevant information (TSPDMAX,etc.) are saved in the ‘OtherMetadata’ section and a processing step with the name of the function is added to the ‘ProcessingSteps’ cell array in the total MATLAB structure.
...
The threshold GDOPMAX is an input variable so it can easily be configured. For each GDOP values failing the test, corresponding U and V data points are replaced by NaN and the corresponding values in the ‘gdopFlag’ and in the ‘Overall_Flags’ ‘overallFlag’ vectors under ‘QCFlag’ ‘QC’ are replaced by 4.
QARTOD recommends two values for the GDOPMAX. GDOPMAX = 10 is used by the National network and GDOPMAX = 1.25 is used for near real time application and web display. HFRNet uses 1.25.
GDOP type used in the cleanGDOP function and the threshold are saved in the ‘OtherMetadata’ section. The GDOP calculated error is saved in the ‘OtherMatrixVars’ section. A processing step with the name of the function is added to the ‘ProcessingSteps’ cell array in the total MATLAB structure.
...
The function cleanTUVQual matches the data points by longitudes and latitudes between the Seasonde total file and the total MATLAB structure. U and V data points with corresponding UQAL and VQUAL values of 999 are replaced by NaN and the corresponding values in the ‘TUVQUAL_Flags’ ‘tuvQualityFlag’ and in the ‘Overall_Flags’ ‘overallFlag’ vectors under ‘QCFlag’ ‘QC’ are replaced by 4.
A processing step with the name of the function is added to the ‘ProcessingSteps’ cell array in the total MATLAB structure, and an index vector and the UQAL and VQAL bad value (999) are saved in the ‘OtherMetadata’ section..
Data Availability
| Include Page | ||||
|---|---|---|---|---|
|
...