Detailed release notes are available on internal wiki (requires login)
November 9
This release was a major update to Oceans2.0: the system was migrated from using an Oracle database to a Postgres database.
Instruments
Updated CORK Parser to parse ASCII Mode Data
- Implemented SCU-V2 Data Acquisition, Control, and Monitoring
- Back End work for Memorial Buoy Data Acquisition
- LBNL Borehole Seismometer FTP job Created
- WERA FTP Jobs created
Automated Testing
- Continued progress on automated DAF testing, task/job automated testing and reprocessing/parsing testing
Earthquake Early Warning
- Architecture for correlator/associator and notification is in progress
- Continuing testing and improvements to EEW data acquisition
Web
The data search page now has a filter column on the side, the user can filter on instrument depth by selecting a depth from and depth to from a set range. they can also enter a time range to filter the device tree by instruments that were deployed in that time range. lastly the device tree now has a visibility filter so what is in the tree reflects what is visible within the current google map bounds.
Infrastructure
- Completed the migration of the metadata database from Oracle to PostgreSQL. This will facilitate performance improvements and is the last key to licensing/commercialization of Oceans 2.0 (Oracle is prohibitively expense, PostgreSQL is open source)
DFO and NOAA projects
- Several new DFO and NOAA related projects are starting up, standby for new features related to this work.
Data Products and Services
- New data product: Time Series Thomson Detided. This is applies to pressure sensors on Bottom Pressure Recorders (BPRs). It applies a specific tide-removing algorithm to make the data more reliable for tsunami detection and for input into tsunami models. The algorithm was supplied by Richard Thomson, Alexander Rabinovich, Isaac Fine and Tania Lado Insua and makes use of Rich Pawlowicz's T_TIDE toolbox.
- Data restrictions have now been applied to all webservices. Continuing work on a UI to manage data restrictions.
October 3 2017
Many new projects are starting up, lots of work in requirements gathering, design. Relatively few new features to report.
Instruments
- Gascoyne E-mail Job improvements
Automated Testing
- Continued progress on automated DAF testing
- Started progress on task/job automated testing
Earthquake Early Warning
- Added de-bias calculation for PPP devices
- Added code to handle scenario of missed or mistaken P-wave detection
- Architecture for correlator/associator and notification is in progress
Web
- Fixed an issue with old SeaTube permalinks
- Fixed an issue where junction boxes would not show devices correctly
- DeviceListing now displays 'misc' information correctly
- Dive comments now allow for larger comments
- Progress on a new Geo-spatial search filter for Data Search
Infrastructure
- Extensive work to migrate metadata database from Oracle to PostgreSQL. This will facilitate performance improvements and is the last key to licensing/commercialization of Oceans 2.0 (Oracle is prohibitively expense, PostgreSQL is open source).
Data Products and Services
- Expanded the enforcement of restrictions on various DMAS sources of data, including a wide range of web services and applications
- Improved ice draft calculation
September 12 2017
Release was delayed due to NOAA dive logging
Performance Improvements
- Moved to https: all services and pages are now https, making the system more secure and compatible with new web standards}
Users may notice that URLs will redirect from http to https for all pages and services
Additional performance improvements for loading webpages
Instruments
- Implementation of WHOI SIIM view, parsers and drivers for WHOI SIIM
Automated Testing
- Adding support for more instruments in automated DAF to improve data quality
PSF App (Pacific Salmon Foundation)
- Added image upload and annotation
Earthquake Early Warning
Improvements/fixes for the client software and notification system, allowing our first client to connect to the system
Infrastructure
- Refactoring, modularization, maintenance
Cache loading improvements
Data Products and Services
- State of Ocean/Env plot fixes and updates
- Progress on ISO 19115 metadata reports
- Data restrictions for all services (in progress)
- Added a new metadata discovery service
- Improvements to the real time data services - scalar and raw data (in progress)
- Started on temporal/spatial filtering in Data Search
August 2 2017
Performance Improvements
- Improved page loading and service response times
Web
- Added a payments system for workshops
Instruments
- DAF testing and support for summer deployments
- Add ports now available via UI
- Camera control via SIIM/node
Automated Testing
- Extensive implementation of unit testing in core/infrastructure modules
- Automated DAF testing
- Search automation: added performance testing and improved reporting
- Service automation: numerous features required to support the new dataProductDelivery services, including JSON parser and prescribed test fields and values, performance testing
PSF App (Pacific Salmon Foundation)
- Various improvements
Earthquake Early Warning
Further improved device simulator for Titan and PPP devices, allowing for synchronized data to fully simulate earthquakes
- Kalman filter automated testing and fixes
- Addition of Mexico and Tohuku data to Automated DAF system to test EEW parsing and calculations
Infrastructure
- Continued work on conversion from Oracle to PostgreSQL (DAO): QA environment now capable of running Oceans 2.0 on PostgreSQL.
- Declarative cache loads for improved cache loading (improves start/restart times)
Data Products and Services
- Further improvements to SOS and dataProductDelivery web services, including additional fields, vocabulary and performance
5 July 2017
Performance Improvements
- Raw data archiving (a core function of Oceans 2.0) has been redirected to the Cassandra NoSQL database, taking the load off of the Oracle database. The Oracle database now only contains metadata. Raw file generation times have been reduced from 14 hours to 2 hours, live archiving speeds have been improved as well and will improve further as additional optimizations are made. This new system is more readily scale-able and easier to commercialize.
Instruments
- DAF testing and support for summer deployments
Automated Testing
- Extensive implementation of unit testing in core/infrastructure modules
- Automated DAF testing
- Search automation: added performance testing and improved reporting
- Service automation: numerous features required to support the new dataProductDelivery services, including JSON parser and prescribed test fields and values, performance testing
PSF App (Pacific Salmon Foundation)
- Various improvements
Earthquake Early Warning
Improved device simulator for Titan and PPP devices, allowing for synchronized data to fully simulate earthquakes
Peak displacement and Peak ground displacement calculations
Sensor value caching for Kalman filter and other calculations
Investigation of alternative messaging protocols to reduce size and latency of data being sent from remote stations
Kalman filter automated testing and fixes
Infrastructure
- Continued work on conversion from Oracle to PostgreSQL (DAO): dev environment now capable of running Oceans 2.0 on PostgreSQL.
Data Products and Services
- Migrate live raw data products to Cassandra
- Update SOS getObservations service to use standard vocabulary
- Improved file size and processing time estimates in dataProductDelivery service
- Improved discovery services in support of the Empower Project dataProductDelivery service
The dataProductDelivery and discovery services have been released internally (can be used by TEF users only), end-user testing is on-going. These services enable users to discover, order and download data programmatically.
5 June 2017
Performance Improvements
- Database SQL performance improvements
Improve performance of Oceans 2.0 Web site(page load, status console,...)
- Improve performance of oceannetworks.ca
Develop a processed files archiving job add support archiving a large amount of files from raw files, postprocess jobs without impacting the normal data acquisition process
- Improve cache management: automatic reload of caches for topology and calibration changes
Instruments
- DAF testing and support for Nautilus cruise
- Declarative parser migration (ongoing)
- Camera system control improvements
Web
- Annotations can display the original author
PSF App (Pacific Salmon Foundation)
- Plot data versus depth, filtering out invalid data
- UI improvements and bug fixes
- Backup data and user info, load multiple quick entry list
Earthquake Early Warning
PPP v1 parser
Kalman Filter
Infrastructure
- Continue work on conversion from Oracle to PostgreSQL (DAO)
Data Products
- ARIS sonar camera movement detection alerts - sends PI an email if a fish swims by!
- Improvements to scalar products in handle edge cases of data gaps with no data rating or sample size, matlab products now report navigation data in-spite of data gap
- The dataProductDelivery service, which enables data product searches to be ordered and downloaded programmatically, has been released to TEF users and will be undergoing end-user testing
9 May 2017
Performance Improvements
- Database SQL performance improvements
Improve performance of Oceans 2.0 Web site(page load, status console,...)
Improve performance of archiver, design a solution for fast multi-day injection of shorestation data
Develop low priority file archiving job to support import of large amount of files without impacting the normal data acquisition process
- Improve cache management(automatic reload)
- Develop performance tests for Web services
Data Products
- Completion of ARIS data products (Manufacturer’s and processed data products - MAT files and MP4 video)
- Support data team to prepare large amount of data products for a local scientist
- Testing and delivery of the alert and monitoring project
Instruments
- Participate in the data acquisition spring cruise project
- Develop drivers for spring cruise
Add declarative parsers to reduce time to develop instrument data acquisition package
- Improvement to Reprocess (generates an e-mail on reprocess error)
Infrastructure
- Prepare conversion from Oracle to PostgreSQL
Refactor Oceans 2.0 code to extract 2 modules (OSGi-fication)
Web
- Refactor Oceans 2.0 code to extract 2 modules
- Web site cruise preparation, Wiring the Abyss (WTA), cruise support
- Add support for multiple simultaneous cruises with data replication
- Support for the Pacific Salmon Foundation project, requirements for new multimedia annotations
- Earthquake Early Warning: Requirements and design (Kalman filter), build drivers
- Requirements and design to support multiple vocabularies NERC
- Design new entry screens and custome grouping of devices to speed up support for new instruments
6 April 2017
Data Products
- Improved Digby and Kitimat State of the Oceans plots
- Added data products for the Wally Structure Light System camera
- Added data products for the Ice Mass Balance Buoy
- Improved workflow for file post-processing
- Improvements to Data QAQC
- Added more data product icons to Data Search
Instruments
- Improvements to Device Workflow Administration Page
- Improvements to Camera Control Page (Driver locking mechanism during a scheduled run)
Web
- Support for User Replication to other Machines
- Added an Expedition User's page for NOAA to manage their annotators
- Caching of JS and CSS file to enhance website loading
- Improve the loading of the Observations page, separate the calls of Navigation Data and Annotations
Miscellaneous
- Completed the Fisheries Science Reporting System for the Department of Fisheries and Oceans (DFO)
- Modularization of code base
- Minor performance improvements to all servlets
2 March 2017
Data Products
- Added DFO Hydrophone data products
- Added State of the Ocean plots (available in Data Preview, replaces and shuts down the VENUS midnight job)
- Improvements to CSV data products (in preparation for a new JSON product)
- Data monitoring, instrument point people, detection and alerts (in progress)
Instruments
- Added support for the ARIS Sonar device
- Improvements to Device Workflow Administration page
- Improvement to Camera Control Page to show scheduled SeaScript jobs (shows next upcoming job, adding a locking mechanism next)
- Improvements to visibility of SeaScripts (now visible from any device that the SeaScript controls)
Web
- Improvements to edge cases in Plotting Utility PNG generation
- Enhancements to the Pacific Salmon Foundation Citizen Scientist app
- Improvements to the layout of the Search Hydrophone Data Viewer
- Glider and NOAA cruise support
- Continuing progress on Earthquake Early Warning
- Prince Rupert Community Page
Miscellaneous
- Properties discovery service now returns flags indicating the properties that can be used in a location search vs primary sensor search.
- Locations discovery service provides more options for filtering, e.g. on data product code
- ERDDAP datasets can be generated for devices with a mobile heading sensor
- Modularization of code base (OSGi-fication)
2 February 2017
Data Products
- Added support for more complex calibration information, which is needed for icListen AF hydrophones in order to generate spectrograms.
- Added support for interleaved dual frequency data for Nortek Signature 55 data products.
- Added support for AZFP Echosounders, to account for the mounting height of transducers
- Data Search and Data Preview plots now include additional information when the "Average" or "Min/Max + Avg" option is chosen. The data rating (how often the data is collected) as well as the averaging interval is now displayed.
- Data Preview now shows Kongsberg rotary sonar daily plots.
- Hydrophone spectrograms are now available for all hydrophones that use FFT format.
- The Kongsberg SMB data product has been improved.
Data Restrictions and Agreements
- Company logos can now be uploaded to be used in data products controlled by data agreements
Dive Logging
- Added several Save options when creating a dive observation:
- Save & Exit - the original Save feature. Returns you to the dive log page.
- Save & Refresh Time - saves the observation, stays on the Observation page, and updates the timestamp. Used when you want to make new observations often.
- Save & Keep Time - saves the observation, stays on the Observation page, but keeps the same timestamp as the observation you just saved. Used when you want to make multiple different observations that refer to the same point in the video.
- Separated dive log tags and quick entries by organization. At the time that a new cruise is defined (now renamed to "expedition"), you define which organization the cruise is for. The choices for observation quick entries and tags is different for all dives assigned to that expedition, depending on the organization selected. At the moment we support two organizations - ONC and NOAA.
- When creating and editing dive log tags and quick entries, these functions have now been separated into separate UI pages under the SeaScribe menu.
Search Hydrophone Data
- Implemented data access agreements, similar to what was added a few months ago to Data Search.
- Access agreements are controlled by the Data Product Permissions tab in User Management
- Access to hydrophone data can be embargoed for a certain date range, or its access can be delayed until it is older than a certain value that can be defined.
- Added a choice of displaying spectrograms produced from WAV files or FFT files (depending on the type of hydrophone).
- Added the ability to specify a date range to view spectrograms. They are still displayed one day at a time but if you use the Next and Previous features it will automatically skip over dates that have no data.
- Added a number of annotation-related features:
- Ability to display the number of hydrophone annotations that apply to each spectrogram time period, with a hyperlink to open a table to view those annotations
- Ability to view all the hydrophone annotations for the currently displayed day.
- Ability to filter the annotations being viewed to show only those containing a specified text string.
- Ability to export the displayed annotations to a CSV file.
- Ability to search a range of dates and only show spectrograms that have been annotated with a specified text string (this filters the spectrograms and other data, as opposed to the feature above that only filters the table of annotations).
Miscellaneous
- On the Device Details page, main tab, a link has been added called Instrument Point People Group. It lists who from each department is responsible for that instrument.
- State of the Environment plots are now updated automatically using a recurring scheduled job.
- Added support for the Campbell Sci Data logger to be installed on ferries.
- Added support for the Pump and Valve Control System (PVCS) for the ferries.
- Added support for autonomous temperature sensors to be used at Endeavour.
- Created a page in Oceans 2.0 (not in oceannetworks.ca) to display real time data for the Campbell River Maritime Heritage Museum. This includes the latest underwater video and sensor readings.
- Continued development of methods to search for data programmatically, including methods to discover where data of a certain type exists if you are not already familiar with ONC's instruments and their locations. This feature is still under development.
3 January 2017
Data Search
- Improved the visibility of "Min/Max + Avg" plots to make it more obvious where the average points are.
- Created scalar profile plots for casts of vertical profilers.
- Created a tabular data product for hydrophones based on spectral probability density percentiles.
- Created data products for the Nortek Signature 55 current profiler.
- Created cast data products for the PSF instruments (Pacific Salmon Foundation or Salish Sea Marine Survival Project) for raw and corrected data versus depth.
- The CSV data product now supports company logos and acknowledgement where data agreements exist.
Instruments and Parsers
- Modified the data parser for the Nanometrics Titan accelerometer used for Earthquake Early Warning to add two additional "derived" sensors for diagnostic purposes. These sensors will give an indication of how close an earthquake came to exceeding the detection threshold, both for the P-wave and the S-wave, to help us tune the detection thresholds.
- Wrote a parser for the EV Nautilus navigation data.
- Wrote a driver and parser for the Hemisphere GNSS V104S GPS compass used on the B.C. ferries.
Hydrophone Viewer ("Search Hydrophone Data")
- Added the option to view FFT-generated spectrograms (called "Broadband Spectrogram" in the Plot drop-down list) where they exist (does not apply to all makes of hydrophones).
- Added the option to view SPD (spectral probability density) spectrograms although they will not be available until later.
- Improved the annotation feature on this page so that a table of annotations that apply to a given spectrogram can be displayed if there is more than one. The list can then be filtered by specifying a text string to filter on, and the table can be exported to a CSV file.
Miscellaneous
- Added deep links to the Data Monitoring tool (in Device Console) so that results can be shared.
- On the Camera page, the Presets section was removed as cameras can now be controlled using SeaScript.
- Improved the Data Product Permissions tab in User Management to better support the administration of data access permissions. For example if the list of access rules is large, it will automatically paginate which greatly improves the speed of opening a large group.
- Started development of the Kalman Filter algorithm for Earthquake Early Warning. This filter and other code will allow us to use data from GPS sensors that will be installed on the land sites in order to improve our earthquake magnitude estimates.
- Continued development of methods to search for data programmatically, including methods to discover where data of a certain type exists if you are not already familiar with ONC's instruments and their locations. This feature is still under development.
ONC Website (oceannetworks.ca)
- Developed a mini website for the ISSMMTC conference (8th International Symposium on Submarine Mass Movements and Their Consequences). See http://igcp640.oceannetworks.ca/
- Modified many of the Community Observatory pages to make them all consistent in appearance, including:
- Shorter introduction with a "Show More" link added
- Consistent Latest Readings variables.
- Consistent titles on data plots
- A photo album to look through photos rather than placing them all on the page at the same time.
- Added an Instrument Health section header with content to be filled in later.
- Did a complete overhaul of the structure of the Installations section of the website, which has been renamed to Observatories. Highlights include:
- New landing page called Expeditions. Content will be added soon, but for now it gives access to sub pages including Wiring the Abyss.
- Installations sub–heading has been renamed to Notices.
- The number of Observatories has been reduced by grouping everything under Arctic, Atlantic, Pacific, and Mobile Platforms.