When you first download data, you can avoid downloading HPF (high-pass filtered) or LPF (low-pass filtered) data by selecting the original data option at the bottom of the data selection page of Oceans 3.0, or by specifying dpo_hydrophoneDataDiversionMode=OD when using the API. If you have an older data set (downloaded prior to May 2019 for most cases, Oct 2019 for others), of if you choose the 'All' option and you want to remove the LPF/HPF files, follow the steps below:

These instructions describe the use of the DOS or Unix/Linux commandline for deleting or moving large numbers of files. For small numbers of files, it may be faster to select files through the UI.

Two approaches are described:

  • delete or move the files directly
  • generate a batch or shell script, and run that script

Program Requirements

For both approaches:

  • Command line, eg.:
    • Command Prompt (Windows)
    • Terminal (Unix/Linux/Mac)

To create, edit, and run a script:

  • Text editor, eg:
    • GVim (Windows)
    • Vim (Unix/Linux/Mac)

Windows v Mac/Unix/Linux

The commands below are for DOS commandline. In Mac/Unix/Linux terminal, the corresponding commands are:

Purpose of commandDOSMac/Unix/Linux
List filesdirls
Create a directoryMKDIRmkdir
Move files to a new directoryMOVEmv
Delete filesDELrm

On both platforms, asterisk wildcards can be used to match filenames with a given pattern, eg. dir *-HPF.* will find all files in the current directory with '-HPF.' in their filename.

In DOS, directory separators are back slashes (\). In Mac/Unix/Linux, they are forward slashes (/).

Deleting/moving files directly on the commandline

  1. Open Command Prompt (Windows) or Terminal (Mac/Unix/Linux)
  2. Navigate to the directory in which your files are located

    1. eg. 

      Change directory (Windows)
      cd C:\foldername\

  3. Move/delete files

    1. To move HPF files to a new directory named HPF, create that directory, and then use the move command, with wildcard asterisks

      Make directory and move files (Windows)
      mkdir HPFFiles
move *-HPF.* HPF\

    2. To delete all HPF files, use the del command, with wildcard asterisks

      Delete files (Windows)
      del *-HPF.*

Generating and running a batch or shell script


Before Starting:

Make sure that there are no spaces in your folder names or file names, as unnecessary spaces will confuse command line. Also, decide whether you would like to delete or move the filtered files. Moving the –HPF files has slightly different steps than deleting them. Steps to move –HPF files instead of deleting them are on the next page.

LPF Steps, moving or deleting LPF files:

LPF data files have the same naming format as raw data files, but are smaller in size.

  1. Sort files by size in file manager
  2. Select and delete or move all abnormally sized files (the size of LPF-filtered files is usually the same as that of regular files but an order of magnitude smaller)

HPF Steps, moving or deleting HPF files:

Both moving and deleting files using a script requires you to generate a script, and then run that script.

In either case, move the file folder to a location you can write the path to easily (in Windows, C: is usually a good choice, or the letter assigned to your external hard drive if the data are saved externally).

Generate a script
  1. Open command line, and change directory to file folder by typing cd path (e.g. cd C:\foldername\)
  2. Type dir *-HPF.* > script.bat /b  and hit enter
  3. Right click list.txt and click “edit with Vim”
  4. Navigate to the top of the page. Click the first character at the top of the page.
  5. Type the following commands…
    Ctrl+q (+ as in press Ctrl and q at the same time)
    Shift+g
    Shift+i
  6. Depending on whether you want to delete or move files
    1. To delete:
      1. Type del path           (e.g. del C:\data\)        and press esc
    2. To move:
      1. Type MOVE path (e.g. MOVE C:\foldername\)      and press esc The filelist should now say     MOVE path\file.extension       for every file (e.g. MOVE C:\data\ICLISTENHF1251_20140104T134650.097Z-HPF.wav)
      2. Repeat step 5
      3. Use arrow keys to move to the end of the first line
      4. Type spaceC:\newfoldername\       and press esc, where space is 1 hit of the space key. The file list should now say           MOVE path\file.extension newpath    for every file (e.g. MOVE C:\data\ICLISTENHF1251_20140104T134650.097Z-HPF.wav C:\data\HPFfiles\)
  7. File>save
Run the script
  1. Doubleclick filename.bat in file folder if windows. If mac, go back to command line and type bash filename.sh
  2. That’s it! All of the –HPF files are now deleted or moved


Explanation of how this works

Goal of Vim editing is to have a text document that reads…

del currentfilelocation\filename1.extension

del currentfilelocation\filename2.extension

.

.

.

del currentfilelocation\filenamelast.extension


OR


MOVE currentfilelocation\filename1.extension filedestination

MOVE currentfilelocation\filename2.extension filedestination

.

.

.

MOVE currentfilelocation\filenamelast.extension filedestination


Each line is a bash command that will delete that individual file or move it from its current location to the destination location. By using the visual block functions of Vim, we can write this command for every individual file without having to type it out as many times as we have files, and save the document as an executable file so that command line will execute each line in succession.


Command explanation glossary

Step 1.

cd – “change directory”


Step 2.

dir *-HPF.* > list.txt /b

“Directory print filename.extension /bare (no additional info other than filename)”


Step 5.

ctrl+q (visual block – allows you to edit based on where the cursor is)

Shift+g (extend the visual block to the full document. Any edits you make will affect that position in every line)

Shift+i (insert text)

If at any point in Vim you make a mistake or get stuck… 
You can click edit>undo to undo a command
You can press esc to get out of different editing modes



  • No labels