This page contains the current year's release notes and new features, primarily for Oceans 3.0, as well as any applications created/maintained by the ONC software development group, including mobile Apps and desktop/PC applications. In progress project status may be presented as well. The information below is written in plain language as much as possible. "Major" releases include all completed items at that time (monthly), while "minor" releases are updates to the previous major release with bug fixes and specific features. Detailed release notes are available on internal confluence pages (internal users only).
March 13, 2023
Minor release: ERRDAP driver fix, MATLAB stuck dead task fix, Axis camera seascript support
Automated geofencing QAQC test (second version based on feedback)
GoogleMap page is removed from O3 menu
Add devicetype information and filter to DeviceListing page
UI message when incorrect deviceid is entered in search field on Device Listing page
Lots of new Selenium tests (e.g., new devicelisting page, ParserDefinition tab of Device Detail page, Data Monitoring tab of Device Console page)
Create scheduled job for processing archiver cql error files - job wont run automatically until more testing is done
Bug fixes
Citation info edge case in data products
Label display issue on Junction Box view tab of Device Console page
Data availability issue in step 2 of data search page for one edge case
Fix bugs in Selenium scripts
Internal projects
External-facing
Java upgrade: QA/production environment (formerly named "Gemini", now "Equinox") has been rebuilt & upgraded (Equinox 3.18.200, Tomcat 6.0.69, other dependencies at latest versions)
Java upgrade: framework startup errors & Tomcat errors (maybe?) will be properly formatted and logged in dmas.log
Improvement: File Management default date range changed to reduce its risk of crashing the JVM
Bugfix: allow taxonomies/taxons with taxonomy matrix attributes and taxon references to be deleted from the UI
Feature: support heap space monitoring on machines other than the web server
Internal only
Java upgrade: support integration testing in Java 8 or Java 17
Improvement: Refactoring of group membership-checking code to simplify use and fix dependency issues
Bugfix: support integration testing on feature branches of Oceans 3
Improvement: Make Oceans Next deployments more resilient to webpack upgrades
Improvement: Automate cleanup of unimportant (but large) files from archived CI builds
Improvement: Deployment scripts will watch for errors logged with serious but rarely-used log levels FATAL, SEVERE, AUDIT during system startup
Improvement: Support "shotgun-style" code reviews in Bitbucket (assignn >= 2reviewers; allow merge after any 2 approvals) with overrides to force full approval of large or critical changes
Bugfix: various UI tests
Improvement: clean up UI test page object initialization
Bugfix: don't force so much re-review on Renovate dependency upgrades
Improvement: validate Jenkins pipelines in branch CI
Community Fishers
Create service to generate processing comment for cast dataproducts in Java
Drop ErddapSensorAttribute table (for CIOOS)
Remove siteDeviceId column from erddapdataset and erddapdataset_hist trigger (for CIOOS)
Remove dependency on Matlab Engine API for Java
Change output of Matlab logs from UDP port to processes output stream
Order of casts in lasso selection tool now newest to oldest
Acoustic Instruments Data Viewer
Convert existing code for Spectrogram widget to utilize Typescript, React hooks & Cypress testing
Convert existing code for Audio Player widget to utilize Typescript
SeaTube
Finished the Deck Log phase one implementation and gave the demo to users (mostly via previous minor release)
End-to-end tests for supporting embedded time code in our video stream
ST web services improvement to support both new Deck Log app and WTA at same time
ST Playlist Improvement due to the resolution flags are changed to null for fixed location video files
Manual test ST for support React 17 upgrade
NOAA on ship servers synchronization tests
Other small improvements
Some minor bug fixes
Earthquake Early Warning
Support sending IRIS meta for our sites when sending live data to ShakeAlert
Add site map to the earthquake catalog page
Fix some bugs with the scheduled jobs populating the earthquake catalog page
OpenAPI
Improve presentation of data product options on the openAPI page
Add the openAPI page to the main menu under more→ web services
get the latest 100 searches, not just those after 2022-12-10
columns are sortable
improved column sizing
Data Policy
New tables and DAL layer for agreement / site device association
Data Products
MATLAB upgrade to R2022b, includes logging system upgrade (log levels and better error handling), matlab-as-a-service custom API that decouples MATLAB version dependency on Java dependency
DOI citations added for netCDF products
Documentation, internal improvements: warning & logging content, unit tests for append_pdfs, netCDF comparator update
Bug fixes: CODAR slowness, citation fixes
Instruments
Quarantine at Archiver: Archiver will now quarantine devices as soon as they are quarantined. Previously the quarantine would only take effect on the archiver after if had processed the backlog.
Download raw file from AD in BPR Import Job
UV Light support in Axis camera driver with seascript commands
[DMAS-75588] - Publication date in citation text is inaccurate
[DMAS-75615] - Captured time is incorrect (which is local time), but after saving the annotation, the timestamp in the annotation table is correct (which is UTC time).
[DMAS-75637] - Take Snapshots not working in SeaTube on QA
[DMAS-75642] - Recording Video not working in SeaTube on QA
[DMAS-75719] - Cast popup closes when attempting to view historical casts on Community Fishers Map
[DMAS-75722] - Cannot Add Resource Role on Device Detail Page
[DMAS-75732] - QAQC and other 4 react pages form is empty in QA
User Story
[DMAS-67558] - As a scientific programmer, investigate and refactor to support latest version of MATLAB (R2022b)
[DMAS-73748] - As a software developer I would like creation of overlapping casts to be blocked at the database level
[DMAS-74165] - As a Data Specialist, I would like modification to automated geofencing QAQC test
[DMAS-74212] - As a software developer I would like to support Matlab version R2022a/b with matlab-as-a-service
[DMAS-74848] - As a software developer, I want to remove obsolete and deprecated code from GroupPrivilegeManager and DmasUserManager so I can streamline development
[DMAS-75274] - As a data steward, I would like the devicetypeid and devicetypename exposed in DeviceListing and update the page
Task
[DMAS-39311] - Disable Equinox logging in /usr/local/gemini/logs
[DMAS-75625] - qaweb2 is logging errors while checking group membership
[DMAS-75699] - Double log of "Acquiring MatlabExecutionEngine for this job run" from MatlabFilePostProcessor
[DMAS-75750] - java_home is set to an invalid directory on ncdeploy
Documentation
[DMAS-75447] - Update documentation for netCDF citation updates
March 2, 2023
Minor release: bug fixes for citation text (searching for date range on archived files causes crashes, fixes for incorrect deployments and missing attributions), annotations, video and hydrophone post-processors
Update Citation in csv data product header to include dataset DOIs
Add one data product icon on Data Search page
Improve search tree loading time (improve generation time, and payload size), affects most search trees
Improve user search on User Management page
Deprecation functionality for Task Management
Automated geofencing QAQC test
Create trx tables for qaqc related database tables
UpdateRdiAdcpComplexParser to ignore invalid data like Negative ArraySize
Fixed topology extension cache so extension Ids should always show up on the topology page (also fixes duplicate extensionIds getting saved)
Calibration formulas now support $null for example: if($77062upto21800m=1,$73462,$null)
Selenium and Cypress tests added/updated
Bug Fixes
Camera video archived with filesource=null
TopologyCache
Min/max qaqc flags for data gaps in csv data product
Generic File Post Processing job
SeaTube
Major features released for Deck Log application
Continue work on Map Layer and WORMS update
Annotation ingestion for Cordell Bank
One bug investigation (missing file resolution flag for some videos)
Data Products
Changes to support an upcoming MATLAB version update: backend java changes, videoQAQC job can now run in current MATLAB version with added error handling
QAQC structure added to time series MAT files containing information and flags for all tests applied to the data
Added a new option to disable (by default) on-the-fly generation of audio formats, this reduces the likelihood of users filling the working space and running slow searches
Minor updates for a plotting and PDF printing edge cases
Community Fishers
Corrected Task Management tree miss-alignment
Corrected blue icons to green for unassigned casts after lassoing and then clearing selection
Corrected odd messaging about FTP transfer in the DataProductDelivery API
Fixed failing UI test in Plotting Utility
Acoustic Instruments Data Viewer
Convert existing code for Spectrogram widget to utilize Typescript, React hooks & Cypress testing
Internal projects
Preparing to run in Java 17 in QA
Update default git branch name ("main" replaces "master")
CIOOS
Data model restructuring for ERDDAP
OBIS Export - fixed issues with event.csv, occurrence.csv and meta.xml content
CKAN
fixed issue with duplicate EOVs
support comma separate list of site device ids in CKAN metadata upload job
ExperiMINTED
Added Search History (PID navigator) prototype page which shows last 100 searches made by a user after 2022-12-10
Instrument search - fix issue with search path not showing in search cart
Earthquake Early Warning
Fix MMI contours on the earthquake detail page, used to be 50% larger than they should be and then as of last month are 100% larger than they should be
Move eewclient and cacheloader queues over to the new jms framework that was created as part of the multiqueue project
Earthquake catalog page now has all earthquakes with magnitude over 3.0 (in area and time of interest)
OpenAPI
The openAPI page can now be accessed through the url /api in addition to /OpenAPI
The openAPI page now has a loading indicator while a request is being processed
[DMAS-73981] - As a data steward, I would like some bugs fixed for the occurrence.csv for OBIS
[DMAS-74012] - As a data steward, I would like some bugs fixed for the meta.xml for OBIS
[DMAS-74691] - Icons for unassigned casts render blue after lassoing and then clearing selection
[DMAS-74886] - Data gaps not marked with correct min/max QAQC flags in CSV minmax avg test cases
[DMAS-74948] - Running a dataproduct search that produces collated PDFs twice locally results in duplicate PDFs being collated into one file
[DMAS-74953] - csv data product doesnt handle data ratings of 0 correctly
[DMAS-74981] - Control icons are missing from the simulator console
[DMAS-74982] - Video widget refresh crashes dashboard
[DMAS-74987] - Junction box alarm settings page is looking for a missing file processing.gif
[DMAS-75070] - Data Search plots (png, pdf) missing data for "Depth" sensor when PU shows data. (work related to Navigation data commissioning work in DATA-4562)
[DMAS-75100] - OSGi tests intermittently time out in Jenkins
[DMAS-75137] - Verification of RemoteCacheReloadServiceActivatorImpl failed on multiple shorestations
[DMAS-75141] - Task Management tree is miss-aligned
[DMAS-75178] - EEW Correlator at eduCloud logging issues with EEWClientActivator
[DMAS-75179] - Gap filling in CSV/JSON no longer respects the sample rate, fills at 1 sec intervals
[DMAS-75186] - Topology ExtID field is buggy - concatenating things that it shouldn't
[DMAS-75193] - Oceans 3.0 gradle builds intermittently fail with an OutOfMemoryError
[DMAS-75243] - Deck Log title is not correct, is only displaying "Screen X of Y"
[DMAS-75244] - Cross-window communication is broken in the Deck Log
[DMAS-75255] - Fixed camera video is archiving with filesource=NULL on a regular basis
[DMAS-75258] - Deck Log link from Expedition Management not working in webpack
[DMAS-75260] - ColumnChooser menu in StatelessTable appears detatched from the table an only opens in the top right corner of Deck Log
[DMAS-75267] - Oceans 3 CI builds sometimes fail with "The message received from the daemon indicates that the daemon has disappeared."
[DMAS-75283] - Instrument search drops path info in cart (was: Instrument search Java CSV data product truncates its DateTo and drops path info in the search cart)
[DMAS-75287] - Couldn't create a new or update an existing annotation for deck log in QA
[DMAS-75288] - The browser tab is closing when reloading the deck log page
[DMAS-75290] - Validation is missing for many cases in the Deck Log's Annotation Entry form
[DMAS-75298] - Attribute value fields in Annotation Entry display incorrectly sometimes
[DMAS-75301] - Add Deck Log page logo same as other pages
[DMAS-75309] - Deck log annotation entry's date validation is incorrect
[DMAS-75312] - Data Search logs an error about a missing image
[DMAS-75316] - video file QAQC no longer detects corrupt video file
[DMAS-75337] - Jenkins Java build agent is out of /tmp space
[DMAS-75366] - Fix error in matlab codarnetcdfpostprocess
[DMAS-75369] - Fix dataset manager to ensure taxons are actually distinct
[DMAS-75371] - Autocomplete is not compatible with Data source selector
[DMAS-75374] - Dates are shown in Pacific time instead of UTC, also dateFrom is displayed for dateTo
[DMAS-75375] - Device Control buttons have flipped gradients
[DMAS-75390] - ErddapManagementTable does not display siteDeviceId or deviceId when accessed via deeplink
[DMAS-75397] - NullPointerException for sensor when reload cache in dev and qa
[DMAS-75434] - Insertion into parserdefinition Table does not trigger an Insert Operation in the parserdefinition_trx Table
[DMAS-75439] - Error On JunctionboxBreakerService in QA
[DMAS-75459] - Oceans Next deployment failed on release/23.01.000
User Story
[DMAS-57229] - As a software developer, I'd like the /DeviceListing page updated to React
[DMAS-69901] - As a software developer, I want to rename the default git branch from "master" to "main" for consistency with git hosting tools
[DMAS-74664] - As a dev, I would like to update DataPreviewManagement Cypress code to fit newer testing-library form
[DMAS-74858] - As a dev, I would like to create a method for deprecating jobs that is visible through the UI
Task
[DMAS-64430] - As a dev, I would like to improve Data Search loading time and reduce it's data overhead
[DMAS-73999] - create a data search icon for the .apt files
[DMAS-74143] - Improvements for user management user search
[DMAS-74440] - Citation line text in CSV data product header replaced with $citation.citation
[DMAS-74465] - Odd messaging about FTP transfer in the DataProductDelivery API
[DMAS-74609] - Create new manual entry form component
[DMAS-74852] - There are duplicate EOVs/ECVs listed in CKAN records when multiple sensors are mapped to the same keyword, we want only distinct EOVs/ECVs added to the landing page
[DMAS-74872] - Start all OSGi bundles in QA and prod
[DMAS-74943] - Update attribution structures and comments in MATLAB data products to use/be compatible with DOI citations
[DMAS-74984] - Remove some unused camera control and data search Javascript and CSS
[DMAS-74986] - Fix Oceans 3 unit tests failing intermittently in Jenkins
[DMAS-75076] - Make basic project templates for persistence bundles
[DMAS-75080] - Disable hydrophone on-the-fly audio data conversion by default with a new option
[DMAS-75081] - Add a QAQC result information to time series scalar MAT files
[DMAS-75085] - Update database script commit process in the README
[DMAS-75118] - Button Set Widget should save its configuration (currently open button sets) to local storage
[DMAS-75120] - Create a communication diagram to show how the widgets in the deck log communicate
[DMAS-75121] - Move Organization web backend to usermanagement
[DMAS-75181] - Add Content Refresh Button to DeckLog Annotation Table
[DMAS-75183] - Reduce Oceans Next coverage temporarily
[DMAS-75185] - create trx table for qaqc, qaqcformula, qaqctestgroup,qaqctestgroupdetail tables
[DMAS-75188] - Add default resource selection for Deck Log to the SeaTubeConfiguration page
[DMAS-75190] - add constraint on im_topologyextension table for unique combination of (topologyid, extensionid)
[DMAS-75191] - Add Auto Reload Menu Option to DeckLog Annotation Table Widget
[DMAS-75201] - As a Data Steward, I need to be able to assign NULL values to sensor data when a condition is met in a JEP function
[DMAS-75221] - Allow users to delete annotations using the Annotation Table Widget
[DMAS-75223] - Allow users to "copy link" to an annotation in a Deck Log
[DMAS-75226] - Add global default resource for Deck Log (No Taxonomy (Expedition))
[DMAS-75235] - Create default layouts for the Deck Log for both logged in and anonymous users
[DMAS-75238] - Saving or updating an annotation from the Manual Entry widget should trigger the table to refresh
[DMAS-75248] - Add organizationId and cruiseId to the payload of CruiseService operation 25
[DMAS-75249] - Update CruiseService call to use operation 25 in the Deck Log so users don't need to be logged in
[DMAS-75250] - Add a cruiseId parameter to the seatube/permissions service so that permissions for deck log annotations can be retrieved
[DMAS-75265] - Load default attributes for taxons when selected
[DMAS-75272] - The "copy link" feature will scroll to and highlight the requested annotation in a Deck Log annotation table
[DMAS-75275] - ONC Lights Driver - Modify remote commands to retry if not successful
[DMAS-75277] - Seascript - add support activateDriver and idleDriver commands for all drivers
[DMAS-75285] - Investigate January 28th missed earthquake detection
[DMAS-75289] - Improve the annotation table's default configuration
[DMAS-75293] - Allow CKAN Upload Job 366 to take a comma separated list of site devices to run with
[DMAS-75303] - Improve Deck Log layout for users with write permission
[DMAS-75304] - Improve the resource fields in Annotation Entry
[DMAS-75308] - Sort the attributes before populate into the form
[DMAS-75311] - Update annotation deletion call to handle cruises
[DMAS-75328] - Make Deck Log layouts the same across all decklogs instead of per cruise
[DMAS-75329] - Fix CSS for background color of deck log
[DMAS-75332] - Remove Dive as a Resource Type from Deck Log's Annotation Entry form
[DMAS-75338] - Align the action icons in the Annotation Table
[DMAS-75354] - Alphabetize data product options from the data product public api service
[DMAS-75395] - Protect the Oceans Next deployment build against changes to npm script names
Implementation
[DMAS-57879] - Update backend to display react version of page
[DMAS-73905] - postprocess Kongsberb SMB files job on demand
Added missing data product options to the open api page for data product delivery service
CIOOS
OBIS Export now fully supports new annotation review schema
Data Policy
Module restructure - moved code to new module, Metadata Products - MP, which houses functionality around generating outputs from metadata, e.g. ISO 19115
ExperiMINTED
Instrument searches now use search_dtl table to enable these types of searches to have a query PID
React 17 Upgrade
Nearing completion, changes include faster builds
Community Fishers
Fixed Date Filter on geospatial map for Webpack
Drop siteDeviceId column from im_iso19115datasetrecord
Community Fishers App
Create CF App Version 3.2.9
Fix errors when building community-fishers in Jenkins
Increased the text size on the data visualizations
Create and test new Community Fishers App compatible with AML Firmware version 3.1.2
Acoustic Instruments Data Viewer
Add UI option for the spectrogramFrequencyUpperLimit dpo in the hydrophone toolbox
Modify dataviewer widget to give accurate dates after shrinking
Data Products
Added new Satlantic Radiometer calibration
Fixes to support upcoming MATLAB R2022b upgrade, including an upgraded logging system:
support new warn once method
find and replace all disp and warning calls with new equivalent in our dataproduct code
implement matlab changes to send logs to UDP socket
Map and test device-level CSV/JSON time series scalar data products for Node and Junction Box devices
Testing and minor changes in support of new min/max options for time series scalar data products
Support instrument searches that use the search_dtl table (so that these searches can have queryPID / DOIs)
Handle erroneous ASL echosounder data (special case where it's firmware was corrupted and it dropped two channels)
Update logos on plots to new ONC blue
Bug fixes: seismometer data products: IRIS webservice bad formatting (problem on their end but we can handle it), Didson bad data