Climatic Research Unit : Research and Projects : SO&P

SO&P: wget information

The wget software/command can be used to make a mirror of a website. By pointing it at the web pages containing the SO&P archive of climate model data, it will attempt to make a copy on your computer of all the files on the SO&P web archive. The SO&P web archive is organised into subdirectories for each model, each simulation and each variable, so that you can control which ones you want to download.

The advantages of using wget are that it only downloads new or changed files. If the download fails part-way through, just start the command again and it will continue from where it had got to. Similarly, if any errors are found in the data archive, and some files are replaced with corrected ones, the same wget command can be used to update just those files that have changed. At least this is how I hope it will work!

Your system may already have wget installed. If not, you can obtain the wget software from:

Download wget source code from http://ftp.gnu.org/pub/gnu/wget/
Native Windows version is available from http://unxutils.sourceforge.net/
If you're interested, full details of wget's options can be found at http://www.gnu.org/software/wget/manual/wget-1.8.1/html_chapter/wget_toc.html

The commands required for each simulation and each variable are available from the SO&P model data web pages - click on the monthly or seasonal fields from the variable list. They all follow this form (command should all be on one line, or use a command-line continuation character, usually backslash \ though for some strange reason \ may appear as the "Yen" symbol in the example below!):

wget --mirror --no-parent --no-directories --accept=nc \
  --http-user=SOAPUSERNAME --http-passwd=SOAPPASSWORD \
  http://www.cru.uea.ac.uk/cru/projects/soap/pw/data/model/echo-g/soaparchive/erik/nc/slp/

SOAPUSERNAME and SOAPPASSWORD can be obtained from Tim Osborn

The switches are:

--mirror - selects options for incremental recursive copying
--no-parent - prevents "leaking" into directories higher in the tree
--no-directories - downloads files into your current directory, rather than creating the sub-directory structure of the SO&P website / web archive. This allows you to create your own structure of directories for organising the files that you download. But if you like the SO&P directory structure, you can omit this option from the command.
--accept=nc - only copy NetCDF files

The exact URL will depend upon the model, simulation and variable. See model data pages for exact commands.

Last updated: April 2004, Tim Osborn