This page contains the current year's release notes and new features, primarily for Oceans 3.0, as well as any applications created/maintained by the ONC software development group, including mobile Apps and desktop/PC applications. In progress project status may be presented as well. The information below is written in plain language as much as possible. "Major" releases include all completed items at that time (monthly), while "minor" releases are updates to the previous major release with bug fixes and specific features. Detailed release notes are available on internal confluence pages (internal users only).
December 19, 2023
Minor release to fix bugs in Jira ticket creation link and CODAR latest preview generation
Improvement: continue rewriting Taxonomy management in Oceans Next
Improvement: remove 30MB of unused & duplicate backend dependencies
CIOOS
Propagate taxon resource changes to other machines' caches
Bug fix: change to value of French translation now updates ISO 19115 correctly
CKAN upgrade to 1.5
Data Policy
Edit dates a deployment is associated with an agreement
ExperiMINTED
Can now re-order tasks within a batch
Bug fix: update DataCite when DOI is tombstoned
Auto-run batches
Earthquake Early Warning
Fixed bug where new magnitude parameters could trigger the epicenter to recalculate if the associator thought multiple earthquakes were happening at the same time. This is a performance improvement in cases where we would have 100+ sites association together and geodetic magnitude parameters coming in.
Fixed magnitude evolution plot on the earthquake detail page
Fixed magnitude parameters not displaying on the earthquake detail page (they were rounding to zero)
API
Add citations to API delivery services (archivefile, raw data, scalar data, data product)
Allow anonymous users to view open API page. If they try to call an API it prompts them to log in.
Added new experimental tables and a scheduled job for summarizing time ranges different data products are available for a device.
Added new returnOptions parameter to the scalar data API service when set toexcludeScalarDatait will not return any scalar data with the service call. Allows users to see metadata such as sensorCodes used to query single sensors.
Community Fishers - Map
Added cast count tooltip/title to map subsite label
Moved the the map's blue pin to the centre subsite
Fixed date filter bug on the CF map
Fixed when hiding the layer of selected casts prevents marker colour change when lasso selection cleared
Corrected ordering of UnassignedCasts and Assigned Casts when lassoing on map
Community Fishers - Misc.
Upgraded storybook to version 7
Fixed concurrency issues with filenamesZipped field variable in SearchResultService
Fixed ONC bathymetries not loading correctly on any leaflet map
Fixed failing plottingutility regression tests
Removed required_columns audit function and associated triggers from QA/Dev - replacing with process improvements
Fixed Data Search Sort/Filter bugs
Fixed and updated automated testing for: Data Search, Plotting Utility and CF Map
Create and map cast profile animation data product
Data Products
Now allow parsing of dataproduct filenames that do not include milliseconds in date
Drop erddapdatasetattribute_erddapdatasetid_fk and remove java references in preparation to dropErddapDatasetAttribute table
Completed the base functionality to support complex data in ERDDAP, this includes:
Investigate metadata and setup
Added soundspeed, and pressure to RDI NetCDF dataproduct
Implemented ability to create RDI NetCDF source ErddapDataset table entry via ErddapManagement
Implemented capability to create datasets.xml section for RDI ADCP data using NetCDF source
Parameterized ErddapCodarNetCdfPostProcessor postprocessor to work with any NetCDF data product
Created task to generate RDI ADCP NetCDF files for use with ERDDAP
Fixed search failure in ISO metadata generation
MATLAB tasks now using tm_task.input for task parameters - batch jobs will now respond to the given parameters
Minor updates: fixed MATLAB Logger class to handle 1D cell arrays properly; created function to replace common check for variable existence and not empty; fixed MatlabFunctionRunner task type not reporting errors properly; improved logging for PAMguard preprocessor
Bug fix for latest CODAR Data Preview
Parser Changes
Attribute parsing now available in all converter types. Attribute parsing refers to parsing a device reading into an attribute value. Previously this was only available when using a CSV converter.
Attributes are now parsed in the "sensors" section not in a separate "attributes" section. Existing parsers that use the "attributes" section are still supported.
All features available when parsing sensors are available when parsing attributes. i.e. field identifiers, preFilters etc.
New prefilters bitmask, bitshiftleft and bitshiftright for binary data.
Prefilter syntax now supports brackets; old format still supported. For example the preFilter "joinData^ ,0,1,3,4,5" can now be written as "joinData( ,0,1,3,4,5)".
Binary prefilters can now be chained. Previously this only worked with string data. For example "bitmask(60)|bitshiftright(5)".
Attribute parsing can be tested in the Parser Definition screen.
Instruments - Drivers
Updates to the RBR zero driver and derived sensors to fix readings for theAZASeafloorPressure and AZACalibrationPressure sensors
Tempo Mini
New lights-only driver.
Standard jmx controls (used by systems to start and stop recording).
Standard seascript lights command plus some custom commands for this camera.
Lights controllable via camera page when metadata added.
[DMAS-75007] - SeaTube V3 goes to random part of dive when switching Quality
[DMAS-77019] - Automatically propagate TaxonResource changes to other machine's caches (was: CKAN and ISO records are not picking up changes in EOV/ECV mappings in QA)
[DMAS-77407] - Plots of LME/N/Z channels on new Guralp Orcus Seismometer(s) aren't right
[DMAS-77710] - The buttons on the ST Annotation List widget header don't work for mobile devices
[DMAS-77857] - SeaTube - sensor graph display not working
[DMAS-78198] - Empty error snackbar after attempting to clone a certain playlist
[DMAS-78222] - Re-pressing Filter does not searches run after first load
[DMAS-78271] - Fix concurrency issues with filenamesZipped field variable in SearchResultService
[DMAS-78279] - ONC bathymetries not loading correctly on any leaflet map
[DMAS-78295] - IllegalStateException when copying clips with the same start date
[DMAS-78303] - Playlist playback reaches weird time-elapsed state if a clip ends and Play is clicked on the video player
[DMAS-78322] - DataSearchRunnerService causing database error on trigger_fct_audit_required_columns constraint
[DMAS-78343] - Styling on video player lost any time a function provided by a context is called
[DMAS-78346] - Dragging and dropping clips in the playlist widget reloads the thumbnail
[DMAS-78358] - PlaylistService is hitting DB constraint when saving an object
[DMAS-78630] - The playlist playback page crashes if files aren't found
[DMAS-78646] - Clip quality options in wrong order
[DMAS-78651] - CODAR latest product searches not completing quickly or successfully
[DMAS-78687] - DMAS Search Failed Notification: 24-Nov-2023 22:05:53 UTC, Burrard Inlet-CF015 (BI-15), Community Fishers ISO 19115 XML Report, SearchHdrId 16845781, UserId 119260
[DMAS-78709] - Earthquake Epicenters are getting recalculated when new magnitude parameters come in
[DMAS-78731] - Sorting on Group column breaks the page
[DMAS-78736] - Deck log and dive log require different numbers of letters to search for a taxon
[DMAS-78747] - NoClassDefFoundError for com.sdicons.json.parser.impl.JSONLexer
[DMAS-78749] - QA: No scalar data from device 24049
[DMAS-78754] - Expedition Management map's cluster markers have undefined labels
[DMAS-78776] - qa fails to load any pages on release branch
[DMAS-78779] - Update function name typos in soundmetricssonarpostprocessjob
[DMAS-78790] - Trigger group children table not showing in correct spot and task rows duplicated
User Story
[DMAS-45972] - We want to move even more jars from onc-deps to repositories
[DMAS-50323] - As a playlist creator, I would like to reorder clips in one of my playlists
[DMAS-74100] - As a software developer, I want to drop support for running Oceans 3.0 on Java 8 in order to allow developers to use new language features
[DMAS-77804] - As a ST user, I want to display more metadata about playlist clips
[DMAS-77971] - As a batch management user I would like to order batch tasks
[DMAS-77975] - As a batch management user I would like to set the order of batch tasks
[DMAS-78068] - As a user, I would like to access any complex data via ERDDAP's NetCDF pathway, starting with RDI ADCPs
[DMAS-78421] - As a scientist, I would like to have a deep link for a specific video clip inside my playlist so that other users can play this clip directly
[DMAS-78652] - As a ST user, I want to be able to group and organize playlists
[DMAS-78654] - As a ST user, I want to be able to search for playlists and sort on more criteria
Task
[DMAS-76631] - Modify matlab code to use taskid within postprocessors
[DMAS-77033] - Sensor Maintenance Page Tabs' React Tables need to change from Modify By to Modified By
[DMAS-77816] - Allow parsing attributes using all standard converters
Minor release: Rayfin driver additional features, API performance improvements, various fixes for DataCite XML, Community Fishers map and lasso, magnitude evolution plot, network console and more.
Returnallreadings from /api/archivefiles by location for sites with many deployments, with small query row limits
Delete user-defined taxons with default attributes
Delete attribute options from user-defined taxons
Improve session persistence across restarts
When clicking Query Annotations button from Device Details or Sensor Maintenance, only open it in one new tab
Data Products
AquaSCAT data products mapped and documented, will be released soon (in-progress)
MATLAB API client library improvements (in-progress)
Community Fisher plot attribution to show role
Seismometer plot range fix: add dynamic scaling if data is out of expected range
Bug fixes: hydrophone search failure due to bad date range on file, sensorcategory deprecated, ffprobe/ffmpeg refactor and bug fix
Data Policy
Support selecting a time-limited portion of a site devicefor association (editing agreement/site device associated time ranges coming soon)
CIOOS
Use file link to ISO in ERDDAP to improvedatasetload performance
ExperiMINTED
Make reason mandatory for a DOI batch
Default sequencing for batch tasks implemented and displayed
Dashboards
Improved table on Management page so it uses the full page and can be searched
Earthquake Early Warning
Set a webserver to receive notifications from Telus
Update the position of PNSN devices when new data comes in
If the second character of the channel code for a PNSN device is an H set the device name to seismometer instead of accelerometer
OpenAPI
Add more information about the ISO 8601 duration date format on the openAPI page
Update the rawdata service to get raw data for the parent if a user queries a piggyback device
Fix the search statistic query so its not calculating seconds and calling them hours. Should result in more accurate estimates from the dataProductDelivery API.
Community Fishers
Added tooltip/title to unassigned cast listing in lasso selection popup
Make use of Task output in Community Fishers Pipeline Jobs
Handle device level searches for Community Fishers ISO 19115 that use new search format.
Refactored the date filter into it's own component
Refactored the help and support request popups into a single file
Refactored the history table and Polygon into their own components
Refactored Geospatial Search page to have a single context for casts Bug Fixes:
DB error when generating a new DOI
Unable to make public Dashboard private
Community Fishers ISO XML no longer lists community and instead lists ONCS as owner of all CF casts
Search bar for Community Fishers Map does not center map on searched location
Instruments
Progress on Rayfin driver improvements
SeaTube
Many new features in Playlists
Design for supporting 10 plus on-ship servers (in-progress)
[DMAS-78313] - Playlist playback shows errors when loading a playlist with clips whose source video no longer exists
[DMAS-78345] - Network Console update hitting DB constraint
[DMAS-78353] - AnnotationService is hitting DB constraint during automated testing
[DMAS-78359] - TaxonButtonSetService is hitting DB constraint when saving an object during automated testing
[DMAS-78365] - Taxonomy page crashes when you navigate to a taxon in qa
[DMAS-78368] - Support Request shows "Failed to load. Try refreshing the page." all the time in QA
[DMAS-78369] - SiteDeviceSubsetMaintenance test failing
[DMAS-78381] - Tens of thousands of errors while processing ActiveMQObjectMessages
[DMAS-78390] - Matlab searches querying database and expecting sensor category ID which no longer exists which is causing all scalar searches to fail
[DMAS-78407] - Both expedition and dive can't be synched between servers
[DMAS-78412] - SearchResultService is returning zip files that are corrupt/not valid
User Story
[DMAS-73146] - As a dataProductDelivery API user, I would like to cancel and maybe restart searches
[DMAS-76361] - Remove OBIS settings UI and old menu navigation from Oceans3 code
[DMAS-76628] - As scientific programmer, I'd like to run matlab code in a scheduled job without a file state or other trigger
[DMAS-76966] - As a ST user, I would like to select video quality when I am trying to playback my playlist, so that I can adapt to different Internet network bandwidth.
[DMAS-76967] - As a ST user, I would like to select video quality when I download my playlist/clips, so that I can adapt to different Internet network bandwidth.
[DMAS-77386] - As a sys operator, limit each user's disc usage on /ftp/pub
[DMAS-77608] - As a ST user, I would like create a new playlist (or add to an existing) by copying certain clips from other playlists, so that I can modify/playback/download the playlist later.
[DMAS-77808] - As a playlist creator, I want to change the quality of a clip in one of my playlists
[DMAS-77936] - As a playlist owner, I want to see a basic thumbnail on my playlists and clips so I can tell them apart
Task
[DMAS-45321] - Move the Cruise persistence layer to an OSGi module
[DMAS-57796] - As an API user I would like to return raw data for piggyback devices (children)
[DMAS-76994] - Improvements for cancelling searches and tasks
[DMAS-77198] - Deprecate the cambridge bay lite page task and page
[DMAS-77465] - ip address or ip port with empty space before/after real value should not be saved
[DMAS-77499] - As a software developer, I would like the Geospatial Search page to use a single context for casts
[DMAS-77562] - Revise header on Dive Log Dashboard
[DMAS-77739] - Add modifydate and modifyby constraint to all tables that contain audit columns
[DMAS-77769] - Copy clips to a new/existing playlist on clicking the Copy button
[DMAS-77776] - Add a quality selector to the Download Clips dialog
[DMAS-77810] - Save the clip quality when saving a clip from SeaTube search
[DMAS-77811] - Set the quality for every playlist_line in the DB
[DMAS-77865] - Create data product entries and mappings for Aquascat products
[DMAS-77874] - Update service to return multiple date spans available to save as new organization agreement resources
[DMAS-77916] - Change DashboardTable to use stateless table instead of sortable table
[DMAS-77921] - Use the reorderable list to implement the playlist clip list
[DMAS-77925] - Update the clip list when a clip name is changed
[DMAS-77927] - Update the video player when a clip quality is changed
[DMAS-77937] - Display the thumbnail of each clip in the clip list
[DMAS-77938] - Display the thumbnail of the first clip of each playlist as its thumbnail on the playlist discovery page
[DMAS-77952] - Add a method to an appropriate web service that returns all the qualities available for a single device and time range
[DMAS-77959] - Java Code change limiting user disc usage
[DMAS-77981] - Add a column to the task type table for default sequence
[DMAS-77982] - Have sequence value set for new batch tasks
[DMAS-77983] - Display the sequence value and set it to be the default batch task table order
[DMAS-77992] - Refactor the history table and Polygon into their own components
[DMAS-78030] - Update front end to accept new service response for site devices
[DMAS-78056] - Update ag_organizationagreementresource table to not allow overlapping dates
[DMAS-78083] - Update reference(s) to SeaTube V3 in Oceans-next
[DMAS-78127] - Replace the url in datasets.xml with a local path to the mounted /ftp/pub folder
[DMAS-78133] - Add a default quality selector to the Edit Clip form
[DMAS-78168] - Create Database Properties for Configuration
[DMAS-78175] - Refactor the help and support request popups into a single file
[DMAS-78180] - Remove the OBIS settings page from Oceans 3
[DMAS-78188] - Use the user-selected quality in the video download task
[DMAS-78189] - Respect user's selected quality when getting the list of clips to play
[DMAS-78192] - Refactor the date filter into it's own component
[DMAS-78193] - email user and some users in the config about the max storage reach
[DMAS-78268] - Make a new servlet and URL for the taxonomy page
[DMAS-78272] - Add a skeleton for the new taxonomy page in Oceans Next
Test
[DMAS-77491] - add open api selenium tests for data product delivery cancel and restart
[DMAS-77866] - Create test cases in QA / search automation for Aquascat data products
October 16, 2023
Minor release (matlab only - no downtime): fix issue with saving very large scalar mat files, fix issue with Nortek Sig55 ADCP device attribute, further improve videofileQAQC job
Minor release: deploy java 17 and fixes for: ADCP plots (scaling and collating), videoFileQAQC job getting stuck and excessive logging, annotation link in Hydrophone Search, java logging, SeaTube filter for bad files and test synchronization.
[DMAS-78066] - Data Preview not showing data preview annotation
[DMAS-78072] - Remove ability to clone to an existing playlist
User Story
[DMAS-74658] - As a NOAA user, I want an option to include OBIS parameters in annotation filters and searches
[DMAS-74760] - As a geospatial map user I would like one source of truth for casts and products
[DMAS-76187] - As a dev, I would like a page to only display Sensor Type information
[DMAS-76901] - As a dev, I would like to save support request to database
[DMAS-76906] - As a Data Steward I want erddapDatasets to automatically be created for applicable casts
[DMAS-76964] - As a ST user, I would like to play/forward/backward and choose any clip to play, so that I can quickly navigate and play my desired clips.
[DMAS-77088] - As a NOAA user of SeaTube, I want an option to export SeaTube annotation search results in Darwin Core format
[DMAS-77163] - As a ST user, I would like to see and edit generic metadata for playlists, so that I can quickly find out accurate information about my playlists
[DMAS-77164] - As a ST user, I would like playlist clips to have a custom name and description so that I can quickly find out accurate information about my playlists' clips
[DMAS-77178] - As a playlist creator, I want to edit basic metadata on clips in my playlists so I can remember more information about them
[DMAS-77209] - please add modifyby and modifydate columns in unitofmeasure table
[DMAS-77519] - As a user, I would like to have a UI page to view SupportRequest records
[DMAS-77523] - As a Data Steward user, I would like to see a column "Color bands" in the Cables table in Oceans 3.0
[DMAS-77546] - Use GCMD taxon/resource mappings to populate ERDDAP keywords for CF CASTS
[DMAS-77557] - As a SeaTube user, I want a standalone playlist discovery page, so that I have more space to navigate and search playlists
[DMAS-77609] - As a ST user, I would like to clone an existing playlist, so that I can modify/playback/download the playlist later.
[DMAS-77770] - As a ST user, I want to be able to see the details of a clip so I know more about it
[DMAS-77778] - As a ST user, I want to download several clips from a playlist
[DMAS-77780] - As a ST user, I want to delete several clips from a playlist
[DMAS-77784] - As a ST user, I want to be able to delete a playlist so that I can manage my playlists better
[DMAS-77785] - As a ST user, I want to be able to download a playlist
[DMAS-77793] - Remove the old playlist management page and update the DMAS menu
[DMAS-77797] - As a playlist creator, I want to be able to edit the name and description of a playlist
[DMAS-77803] - As a ST user, I want to be able to share a playlist
[DMAS-77969] - Add Logout button to Dev Playground
Task
[DMAS-66429] - Improve performance of quarter-scalar-data-generation-historical job to allow for longer term QAQC reprocessing jobs
[DMAS-67696] - Recover sourceSensorId to trustable sensorId for qaqcresult (both prior to and after 2019 implementation)
[DMAS-67865] - which table, sitedevice_hist and sitedevice_trx, is currently used in dmas code?
[DMAS-69883] - Remove tables no longer needed after Budget Module removal in DMAS-58123
Minor release: Aquora lights driver, speed up DOI registration, Data Preview filter update. Bug fixes for Community Fishers, BioSonics data products, CODAR and Nortek Sig55 NetCDF files, SeaTube missing footage, iso file service.
Major release. July release was postponed due to maintenance expeditions, so this release contains more features & fixes than usual.
Maintenance
New Features or Improvements
Deprecate Coastbuster
Remove AnnotationSearch UI page and related permissions (now fully replaced with the new UI)
Email system for Support Request
Several small UI usability update
Add space between id and name for modifyby column on Device Sync page
Change field names from modifyby and modifydate to modifiedby and modifieddate
Add loading indicator on multiple pages (Cables, SiteDevice Listing, QAQC auto tests finder, QAQC auto tests details, sensor maintenance)
Add paging for Page Permissions tab on User Management page
Keep fixed height for header of qaqc auto tests details page at different screen sizes
Add scroll bar for set the process in DeviceWorkFlowAdmin page
Consistent capitalization for sensor data rating uniqueness error message
New button on sensor maintenance tab of device details page to create qaqc auto tests with some pre-filled in information,
QAQC auto tests finder page: ability to display all (include closed) qaqc tests, auto complete device field
QAQC auto tests details page: attributecode is added beside attributeid, auto-complete device field, auto field description field with devicecode,
Hide instrument security Tier from non-ONC users
Update privacy policy link on login and registration pages
Update Help links on Plotting Utility
Add unique constraint on token field in sy_token database table
Drop parserdefinition_hist database table - replaced by a new table
Create Readonly isolated formfield UI page
Display task output on the task management page
Make dayofweek field on task schedule editable on Task Management page
Provide better feedback in an API( response when DMAS system is down
Provide more descriptive error message when no device is found at a location
Restart reprocessing tasks that were in progress when a task machine starts
Fix BPR reprocessing so it can be done across multiple task machines and process files out of chronological order
Get correct list of seascript jobs displaying on camera control page
Fix issue where data acquisition sometimes doesn't work when a device has the attribute useMessageBuffering set to false
Remove date constraint on Dataset Landing Page when search_dtlid is searched
Updates on AutomaticSchedulingJob (doc, name and description)
Lots of new or updated automated tests, many documentation updates, answer questions, support users, etc.
Bug Fixes
Heap issue caused by ScalarDataAPIService
URL display issue for tabs on device details page
Find by name issue on File Management page
Social media links on Digital Fishers page
Sort feature for data rating tab on sensor maintenance page
Sort feature for DateTo column of IP tab on Device Details page
Modifyby display issue on Derived Sensor Parser Definition tab of Sensor Maintenance page
Correct A4 constant value in Seabird43Util
Misspelled task parameter singelton
Internal projects
Bugfix: NetworkConsoleService can't leak secrets
Bugfix: User sessions are more reliably persisted across restarts in Java 17
Bugfix: Web service queries with JSON are properly encoded/decoded in Java 17
Bugfix: A few constraint violation exceptions in web service save methods are reported to the user but not logged
Feature: Simpler internal API for parsing & validating web service parameters
Improvements: Renovate runs much more reliably (automatically updates 3rd party libraries), build improvements
Data Products
Add a 4 hour resample period to the automatic resampling period selection for plots (matches a change in Plotting Utility)
Update data product options on IOS hydrophones to dissuade filling /ftp/pub
Added logging / warning in VideoFileQAQC for conflicting file durations
Bug files: CODAR availability PDFs were coming out as PNGs, add back a tolerance for BioSonics ping overrun, fix Nortek Sig55 data product option documentation link
SeaTube
Feature: Display a countdown timer on the WTA page to the start of the next expedition
Feature: Delete clips from a video playlist
Feature: Download multiple clips from a video playlist (as a single stitched file or as multiple files)
Feature: Display Okeanos turbidity sensors in SeaTube sensor panel
Feature: Search and export deck log annotations from SeaTube Search
Feature: Filter and search annotations by number of reviews and percent positive reviews
Improvement: Dive & DiveListing pages are only visible to dive logger admin users, and only modifiable by system admin users
Fix minor issues in: deck logs, WTA page, UI tests, add more modifyby/date info, improve snackbars, drop unused tables
Community Fishers
Added option to include taskId in matlab calls from MatlabFunctionRunner: enables task/batch specific parameters
Improved logging to narrow down where cancel message disconnect happens
Now only delete search FTP directory from web server for completed user-deleted searches
Fixed using the wrong character for apostrophes in ISO19115 XML Metadata files
Fixed Geospatial Map Casts cannot be downloaded in QA
Fixed layout overflow when selecting multiple casts with the Lasso tool
Created Cypress tests for cast popup components in the geospatial map
Changed formatting on Community Fisher's Geospatial Map search bar
Fixed Clearing the CF date range filter does not restore unassigned cast icons to map
Modified the settings/gear dialog just to do date filtering
Modified the lasso dialog just to do cast selection and downloading
Added new download history icon and dialog to CF map
Added test to geospatial map to check if all casts are re-populated after clearing the filter
CF cast map labels now clickable
Modified the calendar filter icon to reflect if a filter is applied or not.
Fixed Internal service error not fully handled for GeospatialMap Cast Download
Converted siteDeviceSubset_hist to siteDeviceSubset_trx
Converted im_sitedevicesubsetcastdefinition_hist to im_sitedevicesubsetcastdefinition_trx
Acoustic Instruments Data Viewer
Allow default attributes to be added to imported taxons if there new taxonomy is editable
Fixed Audio Widget attempts to fetch URLs from PROD in QA
Fixed Uncaught TypeError is thrown on adding widgets to the Dashboard
Fixed Data Player Widget in QA attempts to load images for unavailable data configurations
CIOOS Pacific
Update citation in ISO 19115 so citations are shown correctly in CKAN (added "https://" to the doi.org URL)
Restore spaces to EOVs in ISO 19115
Fix: Ensure owner is included in ISO 19115 when there is no data agreement
CIOOS & Community Fishers
Support for device type 935
Support CAST ERDDAP datasets in ERDDAP management & datasets.xml
ERDDAP datasets for site device subsets are automatically created
Fix: can not view ERDDAP datasets for a site device when not logged in
JDBC Driver (EJ.3.0.0):
data can be retrieved for CAST ERDDAP datasets
Support sensor ids as source names instead of sensor codes
Dashboards
Loop back on video now supported when playing the Latest Video
We now allow widgets to be any size
Data Preview widget now supports all SOO plots
Text widgets now support links
Fix: full screen video now possible
ExperiMINTED
Search history:
clean up sorting
Improve performance
Fix: Searches created by Hydrophone viewer are now correct and don't crash search history
Batching: Add Batch Task button remains active unless batch is closed
Dataset Landing Page: Full edit support for related identifiers
Internal / minor changes: improved tests and test coverage, snackbar improvements
Instruments, Parsers, Seascript
Bars driver changed to readline driver. Recent changes to firmware have made this driver less reliable. New driver is compatible with older bars driver, not susceptible to future firmware changes.
AML 3 Driver: New driver type for the BC Ferries. (No deployment needed other than MTC for DAF testing.)
Pattern Matcher Parser: New parser type that uses regular expressions. Suitable for non standard data structures (i.e. not csv, xml, json, key/value, etc). Developed for AML 3 data but generalized so it can be used for other types of data.
XML Parser Update: Updated to work fully multi-threaded (needed for multi queue). Can parse attributes and suppors xpath field names if needed. Improved error handling.
Json Parser Update: Simplified configuration. Can use parser features available in other parsers (formulas, some string functions, date parsing). Improved error handling. Documentation in progress.
Seascript random pause command
OpenApi Improvements
SupportISO 8601 Duration timestamps on OpenAPI page
Added copy URL button
Moved clear button to top of page
Speed up location get by tree method
Earthquake Early Warning
Support PNSN devices in both UVic and Kamloops correlators
Add magnitude evolution plot to earthquake detail page
[DMAS-76934] - Wiring The Abyss not showing deck log annotations
[DMAS-76940] - Task Definition Schedule cannot be updated
[DMAS-76953] - Using the wrong character for apostrophes in ISO19115 XML Metadata files
[DMAS-76955] - Eclipse tries to import ca.uvic.dmas.core.system bundles in place of project templates in ca.uvic.dmas.templates/resources
[DMAS-76980] - Page permission deletions failing on User Management page
[DMAS-76987] - Geospatial Map Casts cannot be downloaded in QA
[DMAS-76992] - Requests to ScalarDataAPIService caused heap issues on dcweb2
[DMAS-77042] - Sensor Maintenance Data Rating Does not Sort By Id correctly all the time
[DMAS-77049] - Attempting to sort by 'Date To' if an undefined end date exists results in the table disappearing in the Ip tab of a Device Details page
[DMAS-77050] - Sensor Data Rating Uniqueness Error Message should have consistent capitalization
[DMAS-77051] - List of jobs displayed on Camera page is not correct
[DMAS-77065] - @types packages aren't being automerged by Renovate
[DMAS-77072] - Fix Duplicate Device Issue with IPP Tab of User Management Page
[DMAS-77089] - Instrument Point People Tab cannot add valid device on table page other than the first page
[DMAS-77119] - Change formatting on Community Fisher's Geospatial Map search bar
[DMAS-77121] - Unable to get ERDDAP datasets for a site device when not logged in
[DMAS-77145] - Data Product Option Links under Data Search link to wrong pages for Nortek Signature 55
[DMAS-77149] - Clearing the CF date range filter does not restore unassigned cast icons to map
[DMAS-74271] - Setup a minimum size for Sensor Display widget for an anonymous user
[DMAS-74856] - As an API dataProductdelivery user the request method is not consistent
[DMAS-74901] - Remove minimums on all widgets to make layouts more flexible (was: Make text widget size smaller)
[DMAS-74928] - Update ISO xml metadata to include ONC as "owner" role in addition to custodian and distributor
[DMAS-74997] - Support Windows in Oceans Next's deploy.sh
[DMAS-75824] - As a data steward, I would like an improved way to set the process in DeviceWorkflowAdmin
[DMAS-75879] - Use JAXB for DOIDataset's XML in DMAS
[DMAS-76135] - Implement magnitude evlolution plots into the EQ catalog event page
[DMAS-76412] - Create Cypress tests for cast popup components in the geospatial map
[DMAS-76423] - Convert siteDeviceSubset_hist to siteDeviceSubset_trx and convert im_sitedevicesubsetcastdefinition_hist to im_sitedevicesubsetcastdefinition_trx
[DMAS-76479] - Stop logging unique constraint violations in PhoneManager and WebsiteManager
[DMAS-76483] - Stop logging unique constraint violations in TaxonomyManagerImpl
[DMAS-76484] - Ensure user sessions are correctly serialized through a restart
[DMAS-76489] - Stop completely unnecessary logging of an error message shown to API users
[DMAS-76494] - Assign da_devicequarantine primary keys from a sequence
[DMAS-76952] - Auto-fill the description of new QAQC tests
[DMAS-76957] - Update QA and Dev post database refresh scripts to not email info@oceannetworks.ca when requesting support in those environments
[DMAS-76958] - Update FieldsFilter to remove requiresNetwork
[DMAS-76975] - Front-end: Allow turbidity sensors to display in Sensor Readings panel
[DMAS-76976] - auto fill test description with deviceCode
[DMAS-76979] - create database table to save emails from support request
[DMAS-76988] - clean up database table for coastbuster
[DMAS-77009] - Update seatubeV3/annotations web service to allow for both diveIds and cruiseIds to be provided in the parameters so annotations for both dives and deck logs are returned
[DMAS-77011] - Support export of deck log annotations through the DataProductDelivery service call
Minor release: cumulative update, including: fix for device cache, discontinue service desk replace with support emails, add seascript random duration pause, Data Search delete/cancel fixes, driver & parser updates.
Major release, immediately followed by a minor release (no down time)
Maintenance
New Features or Improvements
Filter feature in batch section of Task Management page
Draggable dialog box on Data Preview Management page
Organization page: trim organization name; add modified by and modified date on dialog box; more clearer message when non-unique organization name is entered
Link to external release notes in Oceans 3.0 menu ← HI!!
Devicetypeid and devicetypename in the general tab of Device Detail page
Deprecate event tab of device detail page
Remove notification fields on deviceaction maintenance page
Denote mobile position sensors in the header of csv & json data products
Upgrade to parallel version of GZip
Improve performance of archive file API getListByLocation
Warning message when create an annotation with datefrom>dateto
Update logo colour
Bump Telnet Console line length to 400 characters
Lots of new or updated selenium tests
Many document updates
Bug Fixes
Incorrect error message when creating new device sync
/dmas in urls in OpenAPI example responses
Typo in alert message on Topology page
Logout issue on Landing page
Caches dequeueing issue on shorestations
NullPointerException in qaqcrestdetailsService
Remove duplicate dmasgroup names
Pamguard tasks sometimes stop unexpectedly
Attempted fix of device cache issue, will put more extensive/complete fix in upcoming minor release
Fixed error onsalinity compensation calculation when run on no data
Correctly set the date_issued value for dataset.xml files
Added createdate column to siteDeviceSubset and siteDeviceSubset_hist tables
Place additional information in pipeline cast data correction and ingestion emails
Acoustic Data Viewer
Fixed QAAudio Widget from pulling prod data
Converted DataPlayerForm to functional React component
Data Products
matlab-as-a-service improvements: better handling of MATLAB fatal errors in postprocess jobs, increase memory monitoring timeout
Citation fixes: attribution end date incorrect, order citation authors alphabetically
Minor fixes:
Improved warning messaging for erroneous hydrophone calibration metadata
Reduce CODAR ruv file hold off time for latest preview searches
Update Nortek Sig55 data products to be handle PLAN0/1 configurations and options
EEW
Various small fitlet improvements
Added 31 PNSN instruments to the event detection algorithm
Expanded our detection area 1 degree East
ExperiMINTED
Search History Pagination
Data Policy
Prototype for associating site devices with agreements
CIOOS / CF
Support saving date created for CASTs for use in ISO 19115
SeaTube
Added support for WTA page in Oceans 3.0. Now live for DFO expedition. Added support for Apple's mobile devices.
In-progress: support more than 10 on-ship servers, playlist management.
Bug fixes, simple UI improvements, added more UI automated tests
Instruments
Axis Camera Driver
New commands: defog, defogLevel, wdr and a new form of takePicture that allows taking a burst of pictures. All new commands are available through SeaScript and the camera web page.
[DMAS-54953] - Parser Definition is not passing batch correctly to DeviceParserDefinitionService
[DMAS-64189] - An attribute management admin user can update duplicated option values
[DMAS-66216] - Unsaved button attribute changes remain even after Cancel is clicked
[DMAS-66856] - Annotation list shows no data after adding the widget back to layout
[DMAS-67048] - Input is not kept for some NOAA dives
[DMAS-75127] - Typo in alert message on Topology page
[DMAS-75448] - Raw file generation is trying to create multiple files with the same name for (maybe) an RBR tiltmeter
[DMAS-75882] - Error snackbar pops out when createing new device synchronization in QA
[DMAS-75916] - Video clips should not show in ST Pro for a fixed location camera
[DMAS-75928] - Log out on Landing page does not work as expected
[DMAS-75932] - PAMGUARD tasks sometimes stop unexpectedly
[DMAS-75964] - Fix Bugs in Data Preview Management Draggable Dialog Boxes
[DMAS-76124] - In PlaylistVideoTest , the snackbar generated from the Copy Link button has an unexpected xpath
[DMAS-76165] - Fix Non-Unique Organization Name Error Message
[DMAS-76195] - Failed to retrieve sensor info error for PVCS View page
[DMAS-76220] - Fix Blank Error-Handling for Organization Name on the Organization Page
[DMAS-76319] - Fix failing tests in SeaScriptManagement
[DMAS-76347] - Improve MatlabFilePostProcessor error handling for the case where Matlab crashes processing a file
[DMAS-76359] - The Menu in Camera Control in DDL Camera appears to have a bug that prevents from changing the parameters also some additional menu functions needs to be added. Laser is not responding to seascript and camera control.
[DMAS-76362] - Attribute Updated snackbar shows when save fails in TaxonomyAttributeManagement
[DMAS-76370] - qaweb2 is logging PSQL errors from ArchiveFileStateDAO.getReadingRangeForDeviceByDate
[DMAS-76409] - COR Files not showing names of communities when downloaded from the geospatial map - bug in citationText for non-DOI case
[DMAS-76446] - Fix nullpointerException geofence test in QA
[DMAS-76458] - Order non-DOI citation owner/author list alphabetically in the DP code
[DMAS-76478] - Duplicate user groups can be created on the User Management page
Remove yellow color from Plotting Utility color schema
Added a new page for maintaining DeviceType metadata
Search hydrophone page UI improvements to prevent users from committing extraordinarily large searches
Dialog box on Organization page now draggable
Support new ferry machine
Hydrophone calibration m-file upload fills out all needed sensitivity vectors
Remove "only run one task machine at a time" text on ReprocessConsole
Change DMAS log level from info to debug for DatastaxScalarDataDAO
Remove IP info on Event tab of DeviceDetail page
Bug Fixes
Improve Data Search and Search Hydrophone page loading times by removing a no longer needed archive query
Google map API key is new generated and implemented in Oceans 3.0 and Oceans-next
Fix pamguard batch page to only show pamguard batch tasks
"Failed to retrieve sensor info" error on PVCS view page
Remove duplicate date in CSV data product for averaging processing type
Internal projects
Fix handling of access restrictions in /api/archivefiles?method=getByDevice
Return an error message when an API service call fails for internal reasons (http 500 error)
Java 17 is ready for testing, this includes various preparation and improvements to deployment processes and scripts
Minor fix to data availability calculations (probably not noticeable to users)
React upgrade: add more automated tests around Leaflet to better test behaviour after upgrade
CIOOS - Community Fishers
Improved maintainability and robustness of ERDDAP
Merge ErddapDatasetService into ErddapManagementService, i.e. ErddapDatasetService has been removed
Refactor datasets.xml generation code
Add support to store and display Site Device Subset ERDDAP datasets
Data Policy
(in-progress) Organization Agreement services to support associating site devices with agreements
ExperiMINTED
(internal) Improve maintainability of Search History page
(in-progress) Add paging support to backend Search History service
(in-progress)More backend support for pi_relatedidentifiers
Data Products
Improve display of sample period data in the header of time series scalar ODV
Add "C" channel for some seismometers and add non-cardinal directions to option set
Add an "index" to connect qaqc final flag values to the qaqc test that contributed that final flag, available in time series scalar MAT files only.
Allow writing MAT files greater than 2 GB (switches to v7.3 MAT files for large sizes), instead of dropping data, also throws lots of warnings. Our MAT files should never be more than ~ 1 GB.
Small changes: live raw data access fixes, refactoring due R2022b upgrade, search status reporting improvements for hydrophone data when data is not available via format conversion, improve backscatter calculation for ADCPs, remove filepostprocessor.class parameter for tasks, java AD file retrieval fix for file modes, exclude QAQC structure except for non-resample time series scalar MAT files.
SeaTube
Added support for WTA page in Oceans 3.0.
Added Oceans Next deployment info in our System Console, so we can tell the remote server versions.
In-progress: Map Layer, supporting more than 10 on-ship server, PlayList management app, UI improvements and tests
Some small bug fixes
Community Fishers
Support multiple fluorometer piggybacks on CF device package
Corrected lassoed cast date filtering on CF map
Parameterized progress indicators/download buttons in cast popup to single component
Acoustic Instruments Data Viewer
Correctedpopulation of toolbox options and limits
Added FLAC Support to Audio Widget with use of HTML5 audio element
Fixed fails to display images for valid data
Earthquake Early Warning
Fix some bugs where earthquake catalog job might sometimes miss an NRCan/USGS detection
Potentially fixed the issue where we lose data during an earthquake detection
Added lls delta trms value to earthquake details page as well as what algorithm epi/mag algorithm was used for each notification
Removed apple push reaction from eew event (it never worked outside of the test phone)
Update event maintenance page to support posting more than 1 MB data (EEW event definition got to large when adding PNSN devices)
OpenAPI
Changes to service parameter values now persist when switching between services
Data product request service parameters dpRunId and dpRequestId are now auto populated based on the response from the request and run services
dataProductDelivery download method now downloads the file rather then showing the file contents in the response view (response view is now hidden)
Make OpenAPI page inaccessible for anonymous users, redirect them to log-in page (all API requires a user identifying token)
Remove some confusing examples in the OpenAPI content
Add OpenAPI wiki link on Landing Page API help button
Add serchresultcomment info in the return of dataProductDelivery
Parser changes are not automatically reverted if attempt to save is not successful.
New button 'Run Tests', helpful for iterative changes to the parser.
When a test failure occurs, the fail reason is displayed. It will show the parser version, test number and fail reason.
Tests do additional checks such as verify test is parsed with the parser version where it is specified and checks if piggyback is configured correctly.
Acoustic Scintillation simulator now has ability to simulate file transfers.
Fixed problem where PVCS event was causing spurious logging in the raw data log.
[DMAS-75612] - make edit organization or create organization panel dragable
[DMAS-75858] - Check all getdeviceattribute usage, add isRequired flag as required
[DMAS-75873] - Add logic to device attribute page for hydrophones for fixing empty entries with less calibration vectors than previous deployments
[DMAS-75927] - Remove filepostprocessor.class parameter from jobs that previously used task type of GenericFilePostProcessor
[DMAS-76017] - Seismic data product SD for high-rate data channel currently unavailble for CH* and EH* channel options
[DMAS-76034] - Clarify for the user "SeaTube Ready" check box - front end
[DMAS-76035] - Add a timeout to cypress component tests in CI
[DMAS-76057] - Support aligned and depth binned data ingestion for new community fishers device package.
[DMAS-76062] - As a ST logger, I would like to have device ID info while I am choosing a device/device data resource in Annotation Entry widget of Deck Log app.
[DMAS-76063] - Parameterize progress indicators/download buttons in cast popup to single component
[DMAS-76065] - Saved Plot in Plotting Utility Does't Save Line Colours, & no Yellow Lines
[DMAS-76083] - Notify a user when their API request fails because of an internal error
[DMAS-76110] - IP information should not show up in Device Listing Event tab
[DMAS-76123] - Email me when Jenkins' docker cleanup build fails
[DMAS-76140] - Round all magnitudes to one decimal place on earthquake catalog
[DMAS-76143] - Add support to the stateless table component to support "fixing" specific columns to the left and right to prevent reordering
[DMAS-76149] - Investigate 2023-04-13 and 2023-04-19 earthquakes
[DMAS-76150] - Run docker prune on all available Jenkins docker agents, not a hard-coded list
[DMAS-76161] - Update SearchHydrophoneData button text
[DMAS-76181] - Adding OpenAPI Wiki to the Landing Page Panel
[DMAS-76189] - Modification of the Earthquake Catalogue
[DMAS-76193] - Add modifyBy and modifyDate fields to the data preview managment dialog
[DMAS-76194] - Change modifyBy and Date to match Modified by and Modified Date column names
[DMAS-76199] - Improve ODV data product sample period and sample size fields
[DMAS-76205] - remove "only run one task machine at a time" on ReprocessConsole
[DMAS-76211] - Hundreds of thousands of unneeded DMAS logs for class DatastaxScalarDataDAO
[DMAS-76221] - Write web service tests for updated CruiseService and DiveListing calls
[DMAS-76225] - Reduce size of post request when updating event definitions
[DMAS-76248] - Handle undeletable .nfs files when cancelling a search
[DMAS-76280] - Update the EEWMQClient version in bitbucket
[DMAS-76294] - Index the final QAQC flag to it's originating test in scalar MAT files
[DMAS-76306] - MAT files handle file sizes larger than GB
Requirement
[DMAS-76072] - Document changed requirements for deck log support
Minor release: logging fixes (to help diagnose problems), fix for videoFileQAQC, fix for archivefiles service not returning the next chunk/page link, remove some inactive EEW subscriptions.
Minor release: support for new hybrid ferries, SeaTube fix, support Community Fishers device packages with 5 piggybacks, archivefiles API fix for unexpected empty response, fix for videoFileQAQC postprocessor, remove yellow from Plotting Utility default colour scheme.
Additional constraints for "Move File" scheduled job
Notify PAMGUARD users of data or forward data to user before data are erased from user directory
Driver start date and last reading added to Device Console page
Replace use ofunpkgreact with internal react library (sometimesunpkggoes down)
Leaflet Upgrade (in-progress)
Bug fixes
"Select All" checkbox doesn't display a tooltip for scheduled jobs
Newly-created task does not appear in task tree
Effective dateTo of QAQC display incorrectly on sensor maintenance page
Citation text in data products (edge cases)
Authentication Keyword Typo in ONC Repository
Community Fishers
FixedERDDAP: cannot retrieve data for JDCB datasets
Created DAL for im_siteDeviceSubsetCastDefinition_hist table
Removed siteDeviceId column from erddapdataset (no longer needed)
Added "light map" link to map and help drop down
Added logging to community fishers data preview generation job
Centered the close "X" not on station search bar
Removed scroll bar from long System Status messages
Fix for empty cor files not setting search result state and to better handle no data cases
AID Viewer
Create Cypress tests for AudioPlayerWidgetConfig
Internal projects
Removed several deprecated & broken servlets
Partly clean up ArchiveFile handling to try to prevent future server-crashing queries
Java 17: improve sanity checks in deployment scripts, prepare QA for upgrade
Several deployment system and developer environment improvements
ExperiMINTED
Related Identifiers
Bug fix: Field drop-downs populate correctly after re-opening Add Related Identifier dialog
Bug fix: Ensure the Dataset Landing Page refreshes correctly when new DOI is searched for
Search History
Add filter for query date range
Improvement in search history query performance
Bug fix: Only the DOI active at the time of the search is displayed in the history
CIOOS
ERDDAP title change - EN/FR text separator changed from "/" to "|" to make it consistent for the CDE CIOOS tool
Data Policy
Site Device Service enhancements to support queries from updated Agreement Licence UI
UI in progress
Instruments
Declarative Parser Changes
New declarative parser type KeyValuePair parser. Suitable for devices that return data as single key/value pair readings or lists of key/value pair readings.
New sensor attribute used in parsing 'ParserField'. This is used in cases where the value returned by a device maps to a different sensor depending on deployment. For example, a device reporting a value for 'port3' may map to a laser in one deployment and a UV light in another. The ParserField sensor attribute allows mapping the reported value to the correct sensor.
New feature for mapping string responses to numeric responses. Some devices return a range of string responses that need to be mapped to numeric values for storage as a sensor value. For example a device reports one of north, north east, east, etc and it needs to be mapped to a numeric value.
Axis Camera Driver
Approximately 15 new commands for changing settings. Commands are available through SeaScript and through the camera control page. Camera control page also displays the current values of these settings.
Take picture and related functions now supported. Commands are available through SeaScript and camera control page. Available resolutions are queried from the camera and displayed in the camera web page.
Earthquake catalog will now display P-wave detections from sites even if we didn't send out any notification for the earthquake
OpenAPI project/page complete
Created job to process hydrohpone .acc files
SeaTube
Start working on Darwin Core export for NOAA but it is paused due to missing Extended Measurement or Fact (EMoF) file
Continue working on Map Layer
Start working on supporting more than 10 on-ship servers
Start working on ST PlayList management app
Finish adding Oceans Next deployment info in our System Console, so we can tell the remote server versions. Need further testing.
Some simple UI improvements
Some small bug fixes
Data Products
Update date created attribute in netCDF and add support for this in search automation
Add java-side time out system for postprocess jobs, remove MATLAB internal timeout, refactor postprocess task parameter (no more need for filepostprocessor.class parameter)
Automated geofencing QAQC test (second version based on feedback)
GoogleMap page is removed from O3 menu
Add devicetype information and filter to DeviceListing page
UI message when incorrect deviceid is entered in search field on Device Listing page
Lots of new Selenium tests (e.g., new devicelisting page, ParserDefinition tab of Device Detail page, Data Monitoring tab of Device Console page)
Create scheduled job for processing archiver cql error files - job wont run automatically until more testing is done
Bug fixes
Citation info edge case in data products
Label display issue on Junction Box view tab of Device Console page
Data availability issue in step 2 of data search page for one edge case
Fix bugs in Selenium scripts
Internal projects
External-facing
Java upgrade: QA/production environment (formerly named "Gemini", now "Equinox") has been rebuilt & upgraded (Equinox 3.18.200, Tomcat 6.0.69, other dependencies at latest versions)
Java upgrade: framework startup errors & Tomcat errors (maybe?) will be properly formatted and logged in dmas.log
Improvement: File Management default date range changed to reduce its risk of crashing the JVM
Bugfix: allow taxonomies/taxons with taxonomy matrix attributes and taxon references to be deleted from the UI
Feature: support heap space monitoring on machines other than the web server
Internal only
Java upgrade: support integration testing in Java 8 or Java 17
Improvement: Refactoring of group membership-checking code to simplify use and fix dependency issues
Bugfix: support integration testing on feature branches of Oceans 3
Improvement: Make Oceans Next deployments more resilient to webpack upgrades
Improvement: Automate cleanup of unimportant (but large) files from archived CI builds
Improvement: Deployment scripts will watch for errors logged with serious but rarely-used log levels FATAL, SEVERE, AUDIT during system startup
Improvement: Support "shotgun-style" code reviews in Bitbucket (assignn >= 2reviewers; allow merge after any 2 approvals) with overrides to force full approval of large or critical changes
Bugfix: various UI tests
Improvement: clean up UI test page object initialization
Bugfix: don't force so much re-review on Renovate dependency upgrades
Improvement: validate Jenkins pipelines in branch CI
Community Fishers
Create service to generate processing comment for cast dataproducts in Java
Drop ErddapSensorAttribute table (for CIOOS)
Remove siteDeviceId column from erddapdataset and erddapdataset_hist trigger (for CIOOS)
Remove dependency on Matlab Engine API for Java
Change output of Matlab logs from UDP port to processes output stream
Order of casts in lasso selection tool now newest to oldest
Acoustic Instruments Data Viewer
Convert existing code for Spectrogram widget to utilize Typescript, React hooks & Cypress testing
Convert existing code for Audio Player widget to utilize Typescript
SeaTube
Finished the Deck Log phase one implementation and gave the demo to users (mostly via previous minor release)
End-to-end tests for supporting embedded time code in our video stream
ST web services improvement to support both new Deck Log app and WTA at same time
ST Playlist Improvement due to the resolution flags are changed to null for fixed location video files
Manual test ST for support React 17 upgrade
NOAA on ship servers synchronization tests
Other small improvements
Some minor bug fixes
Earthquake Early Warning
Support sending IRIS meta for our sites when sending live data to ShakeAlert
Add site map to the earthquake catalog page
Fix some bugs with the scheduled jobs populating the earthquake catalog page
OpenAPI
Improve presentation of data product options on the openAPI page
Add the openAPI page to the main menu under more→ web services
get the latest 100 searches, not just those after 2022-12-10
columns are sortable
improved column sizing
Data Policy
New tables and DAL layer for agreement / site device association
Data Products
MATLAB upgrade to R2022b, includes logging system upgrade (log levels and better error handling), matlab-as-a-service custom API that decouples MATLAB version dependency on Java dependency
DOI citations added for netCDF products
Documentation, internal improvements: warning & logging content, unit tests for append_pdfs, netCDF comparator update
Bug fixes: CODAR slowness, citation fixes
Instruments
Quarantine at Archiver: Archiver will now quarantine devices as soon as they are quarantined. Previously the quarantine would only take effect on the archiver after if had processed the backlog.
Download raw file from AD in BPR Import Job
UV Light support in Axis camera driver with seascript commands
[DMAS-75588] - Publication date in citation text is inaccurate
[DMAS-75615] - Captured time is incorrect (which is local time), but after saving the annotation, the timestamp in the annotation table is correct (which is UTC time).
[DMAS-75637] - Take Snapshots not working in SeaTube on QA
[DMAS-75642] - Recording Video not working in SeaTube on QA
[DMAS-75719] - Cast popup closes when attempting to view historical casts on Community Fishers Map
[DMAS-75722] - Cannot Add Resource Role on Device Detail Page
[DMAS-75732] - QAQC and other 4 react pages form is empty in QA
User Story
[DMAS-67558] - As a scientific programmer, investigate and refactor to support latest version of MATLAB (R2022b)
[DMAS-73748] - As a software developer I would like creation of overlapping casts to be blocked at the database level
[DMAS-74165] - As a Data Specialist, I would like modification to automated geofencing QAQC test
[DMAS-74212] - As a software developer I would like to support Matlab version R2022a/b with matlab-as-a-service
[DMAS-74848] - As a software developer, I want to remove obsolete and deprecated code from GroupPrivilegeManager and DmasUserManager so I can streamline development
[DMAS-75274] - As a data steward, I would like the devicetypeid and devicetypename exposed in DeviceListing and update the page
Task
[DMAS-39311] - Disable Equinox logging in /usr/local/gemini/logs
[DMAS-75625] - qaweb2 is logging errors while checking group membership
[DMAS-75699] - Double log of "Acquiring MatlabExecutionEngine for this job run" from MatlabFilePostProcessor
[DMAS-75750] - java_home is set to an invalid directory on ncdeploy
Documentation
[DMAS-75447] - Update documentation for netCDF citation updates
March 2, 2023
Minor release: bug fixes for citation text (searching for date range on archived files causes crashes, fixes for incorrect deployments and missing attributions), annotations, video and hydrophone post-processors
Update Citation in csv data product header to include dataset DOIs
Add one data product icon on Data Search page
Improve search tree loading time (improve generation time, and payload size), affects most search trees
Improve user search on User Management page
Deprecation functionality for Task Management
Automated geofencing QAQC test
Create trx tables for qaqc related database tables
UpdateRdiAdcpComplexParser to ignore invalid data like Negative ArraySize
Fixed topology extension cache so extension Ids should always show up on the topology page (also fixes duplicate extensionIds getting saved)
Calibration formulas now support $null for example: if($77062upto21800m=1,$73462,$null)
Selenium and Cypress tests added/updated
Bug Fixes
Camera video archived with filesource=null
TopologyCache
Min/max qaqc flags for data gaps in csv data product
Generic File Post Processing job
SeaTube
Major features released for Deck Log application
Continue work on Map Layer and WORMS update
Annotation ingestion for Cordell Bank
One bug investigation (missing file resolution flag for some videos)
Data Products
Changes to support an upcoming MATLAB version update: backend java changes, videoQAQC job can now run in current MATLAB version with added error handling
QAQC structure added to time series MAT files containing information and flags for all tests applied to the data
Added a new option to disable (by default) on-the-fly generation of audio formats, this reduces the likelihood of users filling the working space and running slow searches
Minor updates for a plotting and PDF printing edge cases
Community Fishers
Corrected Task Management tree miss-alignment
Corrected blue icons to green for unassigned casts after lassoing and then clearing selection
Corrected odd messaging about FTP transfer in the DataProductDelivery API
Fixed failing UI test in Plotting Utility
Acoustic Instruments Data Viewer
Convert existing code for Spectrogram widget to utilize Typescript, React hooks & Cypress testing
Internal projects
Preparing to run in Java 17 in QA
Update default git branch name ("main" replaces "master")
CIOOS
Data model restructuring for ERDDAP
OBIS Export - fixed issues with event.csv, occurrence.csv and meta.xml content
CKAN
fixed issue with duplicate EOVs
support comma separate list of site device ids in CKAN metadata upload job
ExperiMINTED
Added Search History (PID navigator) prototype page which shows last 100 searches made by a user after 2022-12-10
Instrument search - fix issue with search path not showing in search cart
Earthquake Early Warning
Fix MMI contours on the earthquake detail page, used to be 50% larger than they should be and then as of last month are 100% larger than they should be
Move eewclient and cacheloader queues over to the new jms framework that was created as part of the multiqueue project
Earthquake catalog page now has all earthquakes with magnitude over 3.0 (in area and time of interest)
OpenAPI
The openAPI page can now be accessed through the url /api in addition to /OpenAPI
The openAPI page now has a loading indicator while a request is being processed
[DMAS-73981] - As a data steward, I would like some bugs fixed for the occurrence.csv for OBIS
[DMAS-74012] - As a data steward, I would like some bugs fixed for the meta.xml for OBIS
[DMAS-74691] - Icons for unassigned casts render blue after lassoing and then clearing selection
[DMAS-74886] - Data gaps not marked with correct min/max QAQC flags in CSV minmax avg test cases
[DMAS-74948] - Running a dataproduct search that produces collated PDFs twice locally results in duplicate PDFs being collated into one file
[DMAS-74953] - csv data product doesnt handle data ratings of 0 correctly
[DMAS-74981] - Control icons are missing from the simulator console
[DMAS-74982] - Video widget refresh crashes dashboard
[DMAS-74987] - Junction box alarm settings page is looking for a missing file processing.gif
[DMAS-75070] - Data Search plots (png, pdf) missing data for "Depth" sensor when PU shows data. (work related to Navigation data commissioning work in DATA-4562)
[DMAS-75100] - OSGi tests intermittently time out in Jenkins
[DMAS-75137] - Verification of RemoteCacheReloadServiceActivatorImpl failed on multiple shorestations
[DMAS-75141] - Task Management tree is miss-aligned
[DMAS-75178] - EEW Correlator at eduCloud logging issues with EEWClientActivator
[DMAS-75179] - Gap filling in CSV/JSON no longer respects the sample rate, fills at 1 sec intervals
[DMAS-75186] - Topology ExtID field is buggy - concatenating things that it shouldn't
[DMAS-75193] - Oceans 3.0 gradle builds intermittently fail with an OutOfMemoryError
[DMAS-75243] - Deck Log title is not correct, is only displaying "Screen X of Y"
[DMAS-75244] - Cross-window communication is broken in the Deck Log
[DMAS-75255] - Fixed camera video is archiving with filesource=NULL on a regular basis
[DMAS-75258] - Deck Log link from Expedition Management not working in webpack
[DMAS-75260] - ColumnChooser menu in StatelessTable appears detatched from the table an only opens in the top right corner of Deck Log
[DMAS-75267] - Oceans 3 CI builds sometimes fail with "The message received from the daemon indicates that the daemon has disappeared."
[DMAS-75283] - Instrument search drops path info in cart (was: Instrument search Java CSV data product truncates its DateTo and drops path info in the search cart)
[DMAS-75287] - Couldn't create a new or update an existing annotation for deck log in QA
[DMAS-75288] - The browser tab is closing when reloading the deck log page
[DMAS-75290] - Validation is missing for many cases in the Deck Log's Annotation Entry form
[DMAS-75298] - Attribute value fields in Annotation Entry display incorrectly sometimes
[DMAS-75301] - Add Deck Log page logo same as other pages
[DMAS-75309] - Deck log annotation entry's date validation is incorrect
[DMAS-75312] - Data Search logs an error about a missing image
[DMAS-75316] - video file QAQC no longer detects corrupt video file
[DMAS-75337] - Jenkins Java build agent is out of /tmp space
[DMAS-75366] - Fix error in matlab codarnetcdfpostprocess
[DMAS-75369] - Fix dataset manager to ensure taxons are actually distinct
[DMAS-75371] - Autocomplete is not compatible with Data source selector
[DMAS-75374] - Dates are shown in Pacific time instead of UTC, also dateFrom is displayed for dateTo
[DMAS-75375] - Device Control buttons have flipped gradients
[DMAS-75390] - ErddapManagementTable does not display siteDeviceId or deviceId when accessed via deeplink
[DMAS-75397] - NullPointerException for sensor when reload cache in dev and qa
[DMAS-75434] - Insertion into parserdefinition Table does not trigger an Insert Operation in the parserdefinition_trx Table
[DMAS-75439] - Error On JunctionboxBreakerService in QA
[DMAS-75459] - Oceans Next deployment failed on release/23.01.000
User Story
[DMAS-57229] - As a software developer, I'd like the /DeviceListing page updated to React
[DMAS-69901] - As a software developer, I want to rename the default git branch from "master" to "main" for consistency with git hosting tools
[DMAS-74664] - As a dev, I would like to update DataPreviewManagement Cypress code to fit newer testing-library form
[DMAS-74858] - As a dev, I would like to create a method for deprecating jobs that is visible through the UI
Task
[DMAS-64430] - As a dev, I would like to improve Data Search loading time and reduce it's data overhead
[DMAS-73999] - create a data search icon for the .apt files
[DMAS-74143] - Improvements for user management user search
[DMAS-74440] - Citation line text in CSV data product header replaced with $citation.citation
[DMAS-74465] - Odd messaging about FTP transfer in the DataProductDelivery API
[DMAS-74609] - Create new manual entry form component
[DMAS-74852] - There are duplicate EOVs/ECVs listed in CKAN records when multiple sensors are mapped to the same keyword, we want only distinct EOVs/ECVs added to the landing page
[DMAS-74872] - Start all OSGi bundles in QA and prod
[DMAS-74943] - Update attribution structures and comments in MATLAB data products to use/be compatible with DOI citations
[DMAS-74984] - Remove some unused camera control and data search Javascript and CSS
[DMAS-74986] - Fix Oceans 3 unit tests failing intermittently in Jenkins
[DMAS-75076] - Make basic project templates for persistence bundles
[DMAS-75080] - Disable hydrophone on-the-fly audio data conversion by default with a new option
[DMAS-75081] - Add a QAQC result information to time series scalar MAT files
[DMAS-75085] - Update database script commit process in the README
[DMAS-75118] - Button Set Widget should save its configuration (currently open button sets) to local storage
[DMAS-75120] - Create a communication diagram to show how the widgets in the deck log communicate
[DMAS-75121] - Move Organization web backend to usermanagement
[DMAS-75181] - Add Content Refresh Button to DeckLog Annotation Table
[DMAS-75183] - Reduce Oceans Next coverage temporarily
[DMAS-75185] - create trx table for qaqc, qaqcformula, qaqctestgroup,qaqctestgroupdetail tables
[DMAS-75188] - Add default resource selection for Deck Log to the SeaTubeConfiguration page
[DMAS-75190] - add constraint on im_topologyextension table for unique combination of (topologyid, extensionid)
[DMAS-75191] - Add Auto Reload Menu Option to DeckLog Annotation Table Widget
[DMAS-75201] - As a Data Steward, I need to be able to assign NULL values to sensor data when a condition is met in a JEP function
[DMAS-75221] - Allow users to delete annotations using the Annotation Table Widget
[DMAS-75223] - Allow users to "copy link" to an annotation in a Deck Log
[DMAS-75226] - Add global default resource for Deck Log (No Taxonomy (Expedition))
[DMAS-75235] - Create default layouts for the Deck Log for both logged in and anonymous users
[DMAS-75238] - Saving or updating an annotation from the Manual Entry widget should trigger the table to refresh
[DMAS-75248] - Add organizationId and cruiseId to the payload of CruiseService operation 25
[DMAS-75249] - Update CruiseService call to use operation 25 in the Deck Log so users don't need to be logged in
[DMAS-75250] - Add a cruiseId parameter to the seatube/permissions service so that permissions for deck log annotations can be retrieved
[DMAS-75265] - Load default attributes for taxons when selected
[DMAS-75272] - The "copy link" feature will scroll to and highlight the requested annotation in a Deck Log annotation table
[DMAS-75275] - ONC Lights Driver - Modify remote commands to retry if not successful
[DMAS-75277] - Seascript - add support activateDriver and idleDriver commands for all drivers
[DMAS-75285] - Investigate January 28th missed earthquake detection
[DMAS-75289] - Improve the annotation table's default configuration
[DMAS-75293] - Allow CKAN Upload Job 366 to take a comma separated list of site devices to run with
[DMAS-75303] - Improve Deck Log layout for users with write permission
[DMAS-75304] - Improve the resource fields in Annotation Entry
[DMAS-75308] - Sort the attributes before populate into the form
[DMAS-75311] - Update annotation deletion call to handle cruises
[DMAS-75328] - Make Deck Log layouts the same across all decklogs instead of per cruise
[DMAS-75329] - Fix CSS for background color of deck log
[DMAS-75332] - Remove Dive as a Resource Type from Deck Log's Annotation Entry form
[DMAS-75338] - Align the action icons in the Annotation Table
[DMAS-75354] - Alphabetize data product options from the data product public api service
[DMAS-75395] - Protect the Oceans Next deployment build against changes to npm script names
Implementation
[DMAS-57879] - Update backend to display react version of page
[DMAS-73905] - postprocess Kongsberb SMB files job on demand
Added missing data product options to the open api page for data product delivery service
CIOOS
OBIS Export now fully supports new annotation review schema
Data Policy
Module restructure - moved code to new module, Metadata Products - MP, which houses functionality around generating outputs from metadata, e.g. ISO 19115
ExperiMINTED
Instrument searches now use search_dtl table to enable these types of searches to have a query PID
React 17 Upgrade
Nearing completion, changes include faster builds
Community Fishers
Fixed Date Filter on geospatial map for Webpack
Drop siteDeviceId column from im_iso19115datasetrecord
Community Fishers App
Create CF App Version 3.2.9
Fix errors when building community-fishers in Jenkins
Increased the text size on the data visualizations
Create and test new Community Fishers App compatible with AML Firmware version 3.1.2
Acoustic Instruments Data Viewer
Add UI option for the spectrogramFrequencyUpperLimit dpo in the hydrophone toolbox
Modify dataviewer widget to give accurate dates after shrinking
Data Products
Added new Satlantic Radiometer calibration
Fixes to support upcoming MATLAB R2022b upgrade, including an upgraded logging system:
support new warn once method
find and replace all disp and warning calls with new equivalent in our dataproduct code
implement matlab changes to send logs to UDP socket
Map and test device-level CSV/JSON time series scalar data products for Node and Junction Box devices
Testing and minor changes in support of new min/max options for time series scalar data products
Support instrument searches that use the search_dtl table (so that these searches can have queryPID / DOIs)
Handle erroneous ASL echosounder data (special case where it's firmware was corrupted and it dropped two channels)
Update logos on plots to new ONC blue
Bug fixes: seismometer data products: IRIS webservice bad formatting (problem on their end but we can handle it), Didson bad data