Climatic Research Unit : Research and Projects : SO&P

SO&P: wget information

The wget software/command can be used to make a mirror of a website. By pointing it at the web pages containing the SO&P archive of climate model data, it will attempt to make a copy on your computer of all the files on the SO&P web archive. The SO&P web archive is organised into subdirectories for each model, each simulation and each variable, so that you can control which ones you want to download.

The advantages of using wget are that it only downloads new or changed files. If the download fails part-way through, just start the command again and it will continue from where it had got to. Similarly, if any errors are found in the data archive, and some files are replaced with corrected ones, the same wget command can be used to update just those files that have changed. At least this is how I hope it will work!

Your system may already have wget installed. If not, you can obtain the wget software from:

The commands required for each simulation and each variable are available from the SO&P model data web pages - click on the monthly or seasonal fields from the variable list. They all follow this form (command should all be on one line, or use a command-line continuation character, usually backslash \ though for some strange reason \ may appear as the "Yen" symbol in the example below!):
wget --mirror --no-parent --no-directories --accept=nc \
  --http-user=SOAPUSERNAME --http-passwd=SOAPPASSWORD \

SOAPUSERNAME and SOAPPASSWORD can be obtained from Tim Osborn

The switches are:

The exact URL will depend upon the model, simulation and variable. See model data pages for exact commands.

Last updated: April 2004, Tim Osborn