Log File
The log file contains our rawest form of instrument data, often called "raw files". These daily files contain all the communications between the instrument and its driver at the shore station, providing an excellent resource for interpreting and trouble-shooting data. Drivers are customized software developed to interact with an instrument. These files are compressed and are often quite large. As a catch-all for all data, these files have no fixed format for their content other than the timestamp and data direction indicator.
The data acquisition process is described in detail in this journal article: https://doi.org/10.3389/fmars.2022.806452 Log files are an essential part of the process. To summarize, raw device data accumulates in a database throughout the day. Log files are compiled daily by the raw file generator task shortly after midnight from that database. This daily compilation process ensures the data is in order and avoid of duplicates. However, data may arrive after the files have been compiled and archived, and that case, the log files are updated by the merge-sort task, although such incidents are rare.
Requests for live or recent data less than 24 hours old data are handled by polling the rawdata service and collecting the data in a temporary version of the raw log files. These on-the-fly generated log files have '_LIVE_' prepended to their filenames and may not match the archived log files due to late arriving data (sorting by time is still done for the _LIVE_ files, but some data may be missing, although this is rare).
Oceans 3.0 API filter: dataProductCode=LF
Formats
Log files are available in TXT format. Every line begins with a timestamp (ISO8601) that indicates the time that the corresponding record was received at the shore station. There are three types of records, which can be identified by the character immediately after the timestamp: '<','>', or ' ' (space).
- The '>' indicates driver-initiated communications (e.g., commands, descriptive text).
- The '<' indicates non-data responses from the instrument (e.g., acceptance of commands, output of metadata).
- The ' ' indicates a data record (e.g., measurements). The format and contents of the data record vary significantly from one instrument to another, depending on the manufacturer and configuration choices. Common formats include comma or tab-delimited text, xml and binary (stored as hexadecimal for readability).
Example: SBECTD37SIP5680_20101006T000000.000Z.txt
Oceans 3.0 API filter: extension=txt
Discussion
To comment on this product, click Add Comment below.