Release Notes

Detailed release notes are available on internal wiki.

Release notes summaries on this page:

• 04 December 2013
• 14 November 2013
• 17 October 2013
• 12 September 2013
• 22 August 2013
• 25 July 2013
• 04 July 2013
• 18 June 2013
• 06 June 2013
• 16 May 2013
• 18 April 2013
• 21 March 2013
• 13 February 2013
• 14 January 2013

2012 Release Notes

2011 Release Notes

2010 Release Notes

23 January 2014 

4 December 2013 

• Launched a new Digital Fishers campaign to identify crabs. The data will be used to test and improve automatic crab detection software being developed by a student at the University of Victoria.
• Made the title of the COVIS Plume Image plot dynamic, according to the data.
• Removed the QAQC and averaging options for Kistler data products on the VENUS data download page.
• Deleted about 23 temporary scheduled jobs that were created for test purposes some time ago and are no longer needed.
• Made performance improvements for certain kinds of MATLAB data products.
• Added documentation for the new Hydrophone Array Spectrogram FFT data product for the Arctic hydrophone.
• Added some automatic quality (QAQC) tests for Arctic sensors. 
• Produced a demo of our new hydrophone visualization tool. The final version will be made available at a later date. Contact us if you wish to see a demonstration.
  
  

14 November 2013

• Quick Plot in Device Console will now plot data for all sensors, even if their data is not currently being archived. 
• Data Search improvements:
  • Added log file and scalar time series data products to AML CTDs.
  • Significantly sped up CSV data product searches for large amounts of data.
  • Added data products for the two autonomous Nortel Current Profilers at Coral Cliff (device ID 23163, 23164).
  • Added a PNG spectrogram data product for the Ocean Sonics icListen hydrophone at Cascadia Basin.
  • Added an FFT spectral data file data product for the Cambridge Bay hydrophone.
  • COVIS data product improvements:
    • Improved the accuracy of COVIS imaging and diffuse plots.
    • Added 3D effect and transparency to the COVIS plots.
    • Added PNG, PDF, and MAT data products for COVIS plume Doppler data.
• Some internal work done to support the eventual switch from Oracle to Cassandra database.
• Added support for archiving sgy (SEG-Y) and wcd file types.
• Added support for achiving data from EK60 instruments.
• Added a new parser for the Falkor thermosalinograph.
• Added a parser for the internal sensors on Ocean Sonics icListen HF hydrophones.
• Implemented a scripting language for controlling instruments, for example for controlling casts on the BPS (Buoy Profiling System).
• Prevented piggyback ports that are not in use from generating error messages.
• Modified the database to support linkage of sensor values with their current latitude, longitude, and depth. This is necessary for mobile devices that do not always stay in one position, such as the Buoy Profiling System (BPS), sensors located on ferries, etc.
• Added a new command into the Paroscientific Nano driver to document the current instrument settings.
• QAQC (data quality) improvements:
  • The user is now prevented from creating a new automatic QAQC test that is identical to an existing test.
  • The QAQC Auto Tests Finder page tables now include columns for deviceid and qaqcid.
  • Implemented automatic QAQC for "derived" sensors (virtual sensors that are computed from the values of other sensors), but only for data collected after November 14, 2013.
  • The Sensor Listing (Sensor Maintenance) page now displays whether a sensor is DMAS-derived (data computations are done in the Oceans 2.0 software) or Instrument-derived (computations are done in the instrument itself). A user with administrative privileges is able to change the Derived Sensor setting on this page.
  • The LastSensorReadings web service now returns QAQC values.

17 October 2013

• Added a new Ocean Data View data product for most instruments. See the "TXT" icon in the Time Series Scalar Data section of the data product selection table. This is a plain text format. It can be viewed in various ways but if you use Excel, set the delimiter to semi-colon.
• Switched from using IBM MQ to Active MQ messaging broker throughout the system.
• Improved the performance of very long CSV data product searches when search by Instrument Type.
• Added a sensor attribute for archiving. For time periods when the attribute is true, data will be archived for that sensor, but otherwise not. This also affects reprocessing.
• Modified several internal web services to start returning QA/QC flags (LastReadingJSON, SensorDataService, LastSensorReadings).
• Created a new format for the quarterscalarsample quarterengineeringscalarsample tables. For now the new and old tables will both be populated as new data arrives but the new tables will not be used yet.
• Added Sony camera to the Cameras menu in preparation for its deployment at Folger Pinnacle next week.
• Changed the SWIP (Shallow Water Ice Profiler) from ASCII mode to binary mode to support backscatter plots (not yet available in the data products in Data Search).

12 September 2013

• The problem with not being able to switch networks or select other sub menus with Internet Explorer 10 has been resolved.
• Device Workflow now supports multiple versions of Processes and Phases.
• Added additional data product types for Kistler accelerometer.
• Changes made to the database to support data from mobile devices such as from instruments located on ferries and the BPS (Buoy Profile System).
• Plotting Utility:
  • Added titles above functional areas to make it easier to use. Eg. "Refresh Plot(s)", "Time Range History", etc.
  • When you first open the Plotting Utility page, a graphic will now appear with basic instructions on how to use this tool.
  • Added a "Hints" link above each plot which will open a pop-up with suggestions on how to use some of the less obvious features.
  • Added more information in the legend, including type of plot (min/max, line, etc), and whether the data had to be downsampled or not in order to fit into the plot area. Also added links to Device Details and documentation (similar to Data Search).
• Data Search:
  • Added automatic resampling to Min/Max plots. Ie. in addition to a choice of fixed resample rates (when there is too much data plot all of it), you can now choose "automatic" which will attempt to pick the optimum resample rate.
  • There is now potentially a different QA/QC flag for the minimum and maximum values of Min/Max plots. 

22 August 2013

Known Issues:

If you are using Internet Explorer 10 and you switch to another network (eg. you go from NEPTUNE to Arctic), you may have difficulty switching back to NEPTUNE again. We are working on a fix. In the meantime you have two options:

• click where the list of networks SHOULD be. Even though the list does not appear, it's still functional.
• Use Internet Explorer 9, Firefox, or Chrome.

Improvements:

•  Data Products in Data Search:
  • Several improvements to the way Min/Max time series scalar plots are displayed, including adding a Min/Max+Avg option, and an option to allow the system to automatically down-sample the data by the most appropriate amount.
  • All barometric pressure sensor values will now be displayed in hPa units.
  • New data products (and drivers) for new instruments being installed soon in Cambridge Bay (Arctic). The instruments will include:
    • an AIS receiver (Automatic Identification System for tracking ship movements), 
    • A VEMCO receiver for tracking tagged fish
    • A Vaisala barometer
    • A new Lufft meteorological station
    • A hydrophone
• Plotting Utility:
  • Added an "All Available" date range, similar to Data Search. This is now the default date range when a sensor is first selected.
  • Moved the legend under the plots to allow room for more details, including links to Data Search, Device Details, and documentation on that instrument.
  • Sensors that do not have any data to contribute to the plot with the currently selected date range will no longer be listed.
  • When zooming in, you can now select the Y axis range as well as the X axis range. You can return to auto-scaling the Y axis by double-clicking on the plot.
  • The vertical size of the plot (not the Y axis range but the actual height of the plot on the screen) is adjustable. Just select the drag bar that appears below each plot and drag it up or down. Note that if you save the plot, it will always be saved at the default height.
  • "Staircase" has been added to the plot properties option list.
  • When sensors from different instruments are used in the same plot, the sensors and their data values are now listed in the Plotted Values table in chronological order. For example if sensor A's data is plotted for the first month, then that sensor is swapped for sensor B and data for sensor B is plotted for the second month, sensor A's data will always be listed before sensor B's data in the Plotted Values table (it used to be random).
• Device Workflow - it is now possible to assign more than one type of process to the same device. In Device Details in the Workflow tab there is now a dropdown list that lets you select the process you want to see the details for.
• A parser was written for the SCIMPI borehole measurement instrument.
• RAS water sampler driver was changed  to accommodate new plumbing.

25 July 2013

Known Issues:

If you are using Internet Explorer 10, in some cases you will not be able to expand the list of locations or instruments in Data Search or in Plotting Utility. If you are having this problem you can fix it as follows:

• Click on the gear wheel at the top right of your browser.
• Select Internet Options
• Select the Advanced tab
• Near the top of the list under Accelerated graphic, check the box "Use software rendering instead of GPU rendering"
• Restart your computer
• Here is a Microsoft article that describes this problem: http://support.microsoft.com/kb/2528233

Improvements:

• Data Search:
  • Added 10 and 15 minute average options.
  • All combinations of raw/clean and NaN/no-NaN are now supported for the CSV and MAT data products. "Clean" means data deemed by the QA/QC tests to be not valid will be removed from the data product and will either be missing or replaced by NaNs (Not a Number).
  • The Data Search Help page has been modified to describe how to use the new data product options feature.
  • The metadata report that comes with most data products now states which options you chose (NaN, clean or raw, resampling), in the Lineage section 
• Plotting Utility
  • Plotted Values popup:
    • Added a disclaimer about the data, plus a link to the same device in Data Search where you can obtain the scientific data.
    • Added a row to identify the fields in the table (Time, Min, Max, Avg, etc)
    • Removed the list of possible averaging time periods and replaced it with either "Down-Sampled Data" or "All Data (not down-sampled)", depending on whether the date range was small enough to display all data points without averaging.
    • When the display type is "MinMax" (but not if it's MinMax+Avg), the table also shows the difference between the min and max values for each point.
  • Also added a link to Data Search beside the legend and in the legend.
  • If the plot includes data from more than one instrument at different times, each instrument will be plotted in a different colour. This can only happen if you sort by location.
  • In recent releases Plotting Utility was only plotting data if at least 70% of the samples in each averaging bin passed quality control. Starting with this release Plotting Utility will again show all data regardless of quality control. In the future this feature will return along with the ability to set the threshold yourself.
  • When creating a new plot where there is no data, instead of nothing appearing, you will now see an X and Y axis. This should make it more clear that things are working correctly but you have selected a date range where there is no data.
  • Improved the way the Y axis is auto-scaled so it's re-evaluated every time the date range changes.
•  Added the ability to sort by date and time in the Dive Listing and Dive Log tables.
• Added a zoom function for the Tempo-mini camera at Endeavour.

4 July 2013

Improvements:

• Data Search – a new data product options selection form appears when you select a data product. 
  • Make the sections you want, then Add to Cart. If you want more than one item in your cart before you process them, it is not necessary to press Add to Cart one at a time, just at the very end. 
  • If you have more than one item checked, the options form applies to the check box highlighted in green.
  • The data product options are different for different types of data products. For instance PNG plots has different options than CSV.
  • "CSV NaN" option has been removed because now you can select CSV and then select "NaN" in the options form.
  • "QA/QC" (data quality) has now been enabled in some of the data products. These data products allow you to choose between "clean" and "raw". 
    • "Clean" means data that is deemed to be bad according to the QA/QC tests will be removed from the data product. It will either be removed entirely or replaced by NaNs (Not a Number) according to your selection. 
    • "Raw" means all data will be reported; if the QA/QC tests believe it's bad or suspect data, an appropriate flag number will be put beside that record. In the case of plots, coloured flags will be put beside the bad data (one flag for each group of bad data, not one flag per data point) and the suspect data points will be red or yellow.
  • For MAT and CSV, all data will be "raw" at this time, except for averaged data, which is always "clean". Raw and clean options will be enabled in a future release.
  • The "clean/raw" option is available for all the Data Search plots (PNG, PDF, staircase, profile, etc).
  • For more details see the link at the top of the appropriate data product checkbox. Eg. Time Series Scalar Data or Time Series Scalar Plot
• Plotting Utility:
  • The High/Low display option has been renamed "MinMax".
  • A new "MinMax+Avg" option has been added. This is now the default for all new plots (but not for saved plots). As usual, the display type can be changed using the Plot Properties feature.

• LISST driver changed to support bioblock.
• RBR Virtuoso parser change.
• Improved ADCP driver when slaved to Biosonics devices.
• Newer mechanism for ZAP/ADCP driver synchronization.
• Junction box labels replace breaker #s in JB View, JB Alarms, Quickplots, and Plotting Utility.
• A serial number attribute was created for sensors.
• JB View has been removed from the global menu (must access from Device Console).
• The Device Console diagram is network agnostic. Ie. it doesn't care whether the devices are in the NEPTUNE, VENUS, or Arctic observatories.
• The devices in the Device Console / Junction Box tab are now ordered by junction box number.
• Added a WAV file data product for the Ocean Sonics icListen LF hydrophone.
• Enabled data products for the Sony Outreach camera at Folger Pinnacle.
• Better support for data products for devices that do not produce data on a regular basis, such as camera pan and tilt controls and lights.
• Added "EC -Speed of Sound" as a device attribute for RDI ADCPs.
• Added additional configuration commands to the ADCP driver.
• Added a driver for the RBR Global BPR.
• Added additional features to SWEP - Sensor Web Enablement Platform.

18 June 2013 (minor release)

ONC Improvements:

• Implemented the initial stages of SWEP - Sensor Web Enablement Platform. This will eventually allow users to query scalar data following the Sensor Web Enablement family of standards promoted by the Open Geospatial Consortium. The initial step was to develop the Sensor Observation Service (SOS). This release is only to verify standard compliance.

NEPTUNE Improvements:

• In Device Console, in the Junction Box tab, the J numbers now map correctly to breaker numbers.
• For the Tempo Mini instrument at Endeavour, the configuration was updated to handle the new lights.
• Added a lights driver for the Sony camera at Folger Pinnacle.
• Added extra configuration commands to the RDI ADCP driver.
• Added "EC - Speed of Sound" device attribute to RDI ADCPs.
• Wrote a parser for RBR BPRs.
  

6 June 2013

NEPTUNE Improvements:

• The Cancel button has been removed from the Reprocess Console.
• Barkley Canyon MidEast POD3 has been renamed Barkley Canyon MidWest POD3. MidWest POD4 has been renamed to MidEast POD4.
• Changed the QA/QC (data quality) threshold for averaged data from 80% to 70%. Ie. if less than 70% of the records for a given averaging "bin" (one minute, one hour, or one day) passed quality control, none of the data in that bin would be used.
• Improved how VPS (Vertical Profile System) "cast" data products are computed in order to use resources more efficiently.
• Added the ability to search and download certain information for the new Sony camera at Folger Pinnacle using Data Search, such as MP4 video, log file, and plots showing when the pan and tilt controls and the lights have been operated.
• Added QA/QC (data quality) flags to the following Data Search data products:
  • plots for the Bore Hole temperature probes
  • CSV data product for the VPS
  • plots for the A.G.O temperature probes
  • scalar profile plots for instruments on the VPS
    

16 May 2013

NEPTUNE Improvements:

• Added QA/QC (data quality) to more of the data products, such as VPS casts, and adding QA/QC flags to data search plots.  
• Added a "count" column to CSV and MAT data products to show how many "good" samples were available for each of the resample bins, based on quality control ("QA/QC") tests.
• Metadata report now matches data products in terms of dates data are available (contents and filename).
• Added thirtysecond-engineering-summary scheduled job
• Modified Kongsberg sonar driver to support master/slave mode.
• Added pan and tilt support for new cameras being deployed soon.
• Whenever a calibration formula is created, changed, or deleted, added a warning that the data must be reprocessed for the changes to take effect.
• Now allow a calibration formula to to be updated as long as it has a different starting date (DateFrom) from all other formulas for that sensor. Also allow a formula to be deleted.
• The new Sample Size field in Device Details can now be edited. This field will eventually be used to better handle instruments that send data in bursts rather than evenly spaced data, such as some of the sonars.
• Device Console now shows all instruments from all networks at the same time without needing to change networks.
• Some back-end work done to support QA/QC (quality control) in data products. You will soon (in a future release) be able to select whether you want your data products to contain raw or cleaned data based on quality control information.
• Plotting Utility now leaves gaps in line plots where there is no data. Previously the gaps had been bridged with a straight line.
• Added support for dive videos taken by Ocean Explorer ROV.

VENUS or ONC Improvements:

• Added support for Kistler accelerometer
• MacArtney Winch - added Command and Control Functionality. 
• BPS (Buoy Profile System) - Added MacArtney Winch to Driver Schedule.
• Wrote AML CTD driver and parser
• Wrote new driver for daisy-chain Imagenex devices.
• Wrote driver and parser for Honeywell HMR-3300 orientation sensor.
• Wrote driver and parser for Omegabus thermistor chain.
• Added WAV and PNG data product options for the IC Listen hydrophone
• Added monthly scatter plots for ferry data.

18 April 2013

NEPTUNE Improvements:

• Implemented "QAQC" in scalar data products. Data will start to be analyzed for invalid values and flagged. It will take some time before all instrument data is being analyzed, but the infrastructure is now in place to support it.
• Data can be analyzed "manually" (by hand) or automatically by software.
• Automatic tests include (not an exhaustive list) out of range tests, spike tests, high gradient tests, and stuck value tests.
• Scalar data products (CSV, MAT, PNG, PDF) now show a QAQC flag for each record. CSV and MAT list the flag value, ranging from 0 to 9.PNG and PDF plots highlight potentially invalid data with yellow or red colours, and with a yellow or red flag in the center of each group of flagged data.
• Flag values: 0 means no analysis has been done, 1 means good data, 2 - probably good, 3 - use with caution, 4 - bad data, 7 - averaged data, 8 - interpolated value, 9 - data gap. See here for more details: http://venus.uvic.ca/2011/05/qaqc-for-venus-scalar-data/&nbsp
• In a later release the user will have the ability to select "clean" or "raw" data in their data products. For now all non-resampled data is "raw", meaning that invalid data is still present but is flagged. All resampled data is "clean", meaning that invalid data has been removed; for the CSV NaN option it will be replaced with "NaN" (not a number); for the other options it will be removed entirely.
• In the case of averaged data, if less than 80% of the expected data values for a given averaging point are not present (either due to data gaps or failing a QAQC test), that averaged point is treated as a failed QAQC data point.
• Data Search - "sort by Instrument" now has an additional choice level. Besides selecting an instrument you now also have to select where that instrument was deployed. Note that "MTC" means Marine Technology Center, our on-shore testing labs (which in some cases have data). The data availability graph will only show data obtained from the selected deployed site. Regardless of the data range you select, the data products will only contain data from your selected site. If you want to see data from a given site regardless of which instrument was deployed, use "sort by Location".
• Plotting Utility - has the same "sort by Instrument" changes mentioned above.
• Data Search - the "min/max" resampling option has been temporarily removed. It will return in a future release but only for certain data products.
• Improvements were made to the internal "Device Workflow" tool.
• Modified SensorDataService, DeviceAttributeService
• Added more information in search failure e-mails if an Instrument search.
• Improved the performance of Data Search for generating matlab scalar data products by up to 3X.
• Modified the Nortek data products to detect velocity-phase wrapping.
• In CSV files that reference data from more than one instrument (such as on the VPS), it is made more clear which sensors are external to the main instrument.

VENUS Improvements:

• Added TEOS calculated variables to data products.
• Added 1, 10, 15, and 60 minute averaging options to data products.

21 March 2013

General Improvements:

• Live camera videos linked to the NEPTUNE Canada web site can now be played on an iPad, iPhone, and BlackBerry Playbook.The same is true for the DMAS (Oceans 2.0) "Cameras" pages. This will be enhanced in future to also support SeaTube and Digital Fishers.
  

NEPTUNE Improvements:

• Plotting Utility legend and plot image details now also specify the location of the instrument being plotted. A title is now automatically generated as well, with the same information (you can always edit it using Plot Settings).
• In Plotting Utility, when averaging is required in order to reduce the number of displayed points (which happens in most cases), the timestamps are now located at the center of the time bins.
• In Data Search where "average" type resampling is selected, the timestamps are now located in the center of each time bin.
• When doing Data Searches where the metadata reports are automatically generated, the processing time to produce the report has been significantly reduced, which also means that you will get your data products sooner. In most cases the metadata report should be ready less than one minute after the data product is ready.
• "QAQC" has been partly implemented in order to automatically (or manually) check for incorrect data values, such as values that are out of the expected range of the instrument, or that have sudden spikes that could not have natural causes. So far the only data products that are affected are CSV and MAT Data Searches for Resampled data where the average is 1 hour or 1 day. Also affected are Plotting Utility plots where the data is averaged over 15 minute or longer periods (which can be confirmed by checking the Plotted Values table). In these cases, if one or more QAQC tests failed for one or more data samples, and if less than 80% of the samples needed to compute a single averaged point did not pass QAQC, then this point will not be used in the data product or the plot. Additional data products will use QAQC in future releases.
• Internal users are now able to create "station" level QAQC tests.
•  Improvements made to the experience a new user has in creating an Oceans 2.0 account, and existing users have in logging in, such as:
  • Making it more clear which fields are mandatory in order to create an account.
  • Making error messages larger and moving them to be directly above the registration form where they will be more obvious.
  • Retaining as much information as possible if you fail to register, so you only have to correct the cause of your problem (such as mismatched passwords).
  • Logging the cause of registration failures (such as mismatched passwords, invalid e-mail accounts, missing mandatory fields, etc) so that we can make further improvements if necessary.
• Added a Workflow tool (internal users only) to help us track the workflow of new instruments from ordering to deployment and commissioning.
• Added a hyperlink to the digitalfishers.net blog in the Digital Fishers campaign popup.
• Data Search data products involving instruments where we have changed the rate at which they collect data have been improved. If you ask for a date range that spans a period where the sample rate ("Data Rating") changed, will now give a more accurate estimate of the number of samples expected. CSV and MAT data products will no longer create new files when the data rating changes, but will list all data ratings and the date and time these change in the file header.
• Improved the processing time to produce a "time series scalar profile plot" data product for VPS ("Vertical Profile System") devices.
• Added a feature for internal users to either allow QAQC results to be overwritten when doing reprocessing, or not.
• Added drivers for AML Oceanographic CTDs
  

VENUS Improvements:

• Added ASL AZFP driver
• Added driver for Paroscientific Intelligent Digicourse pressure sensor.
• Added support for upcoming deployment of BPS (Buoy Profiling System)
• Added option to download oxygen sensor data in umol/L units.
  

Arctic Improvements:

• Made improvements to the driver for the Cambridge Bay WETLabs Water Quality Meter
  
  
  
  

13 February 2013

NEPTUNE Improvements:

• SeaTube - the Cruise and Location branches (on the left) now open automatically to the first level of the search tree. This should make it more clear how to select a video to be played.
• Data Search - metadata reports are now automatically provided for non-scalar instruments (but still only available on the NEPTUNE network at this time).
• Data Search - the dotted line on the map has now been labelled "Line P" to make it more obvious that this shows the locations (in general) of the Line P buoys. Clicking on the label brings up more details.
• Data Search - Subsampling:
  • "Subsample Type" has been renamed "Resample Type"
  • Resample Type is now a hyperlink to a Wiki page that explains how the resampling is done.
  • The "Average" resample  type is now computed differently in some cases. For details see the Resampling Wiki page at Resampling
• Data Search - the PNG and PDF plots now use a "heat map" method when the number of points is too large to be plotted. The heat map plot will show a distribution of points, with a darker colour where there is a higher concentration of points.
• Data Search - now support one minute average resampling for A.G.O. temperature probe data.
• Data Search - modified the CSV data products, when using resampling, to set the timestamps at the middle of the resample periods rather than the beginning.
• Created an external "dataproduct" web service  to look up data products and data product formats.
  * Created the "stations" web service to look up station code and DeviceCategory.
• Running tasks can now be cancelled (if necessary) using Task Monitor.
  

14 January 2013

NEPTUNE Improvements:

• All SeaTube videos now play as soon as they are selected (previously you had to sometimes select a point further down in the dive log before the video would play).
• SeaTube resolutions are now called Low, Medium, and High. (Low displays Satellite footage, Medium is appropriate for consumer broadband, and High requires high bandwidth connections)
• Decimation sub-sample option was removed from Data Search until an anti-aliasing problem can be resolved.
• Get a warning in SeaTube if flash player not installed, and a link to install it. Videos won't play without the flash player.
• Added PDF versions of the "cast" plots for the VPS (vertical profiler system). 
• Improved the appearance of some of the PNG plots in Data Search.
• Added Stuck Value, Spike, and Gradient QAQC tests to prepare to implement QAQC analysis of NEPTUNE data soon. Also added an automatic e-mail feature (to NEPTUNE staff) if any QAQC test fails.