wget Command Cheat Sheet

wget is a free utility for non-interactive download of files from the web. It supports HTTP, HTTPS, and FTP protocols, recursive downloads, conversion of links for offline viewing, and can operate in the background even when the user is not logged in.

Synopsis

wget [OPTIONS] [URL...]

Description

GNU Wget is a command-line utility for downloading files from the web. Unlike curl, wget excels at recursive downloads, mirroring websites, and downloading in the background. It's non-interactive, meaning it can work in the background while the user is not logged in.

Basic Usage

Download Single File

wget http://example.com/file.zip

Download and Save with Different Name

wget -O custom-name.zip http://example.com/file.zip

Download to Specific Directory

wget -P /path/to/directory http://example.com/file.zip
````

---

## Download Management

### Resume Interrupted Download

```bash
wget -c http://example.com/large-file.iso

The -c option continues getting a partially-downloaded file.

Download in Background

wget -b http://example.com/file.zip
# Check progress
tail -f wget-log

Quiet Mode (No Output)

wget -q http://example.com/file.zip

Show Progress Bar Only

wget --show-progress -q http://example.com/file.zip

Limit Download Speed

wget --limit-rate=200k http://example.com/large-file.iso
wget --limit-rate=1m http://example.com/file.zip

Set Number of Retries

wget --tries=10 http://example.com/file.zip
wget -t 0 http://example.com/file.zip  # Infinite retries

Timeout Settings

# Connection timeout
wget --timeout=30 http://example.com/file.zip

# DNS timeout
wget --dns-timeout=10 http://example.com/file.zip

# Read timeout
wget --read-timeout=20 http://example.com/file.zip

Multiple Downloads

Download Multiple Files

wget http://example.com/file1.zip http://example.com/file2.zip

Download from File List

Create urls.txt:

http://example.com/file1.zip
http://example.com/file2.zip
http://example.com/file3.zip

Download:

wget -i urls.txt

Download with Wildcards (FTP)

wget 'ftp://ftp.example.com/directory/*.pdf'

Recursive Download

Basic Recursive Download

wget -r http://example.com/directory/

Limit Recursion Depth

wget -r -l 2 http://example.com/directory/  # 2 levels deep

Mirror Entire Website

wget --mirror --convert-links --page-requisites --wait=1 http://example.com

Shorthand:

wget -mkEpnp -w 1 http://example.com

Options explained: - -m (--mirror): Turn on mirroring options - -k (--convert-links): Convert links for offline viewing - -E (--adjust-extension): Add .html extension to files - -p (--page-requisites): Download images, CSS, etc. - -np (--no-parent): Don't ascend to parent directory - -w 1 (--wait=1): Wait 1 second between requests

Download Specific File Types

# Download all PDFs
wget -r -A pdf http://example.com/papers/

# Download multiple types
wget -r -A pdf,docx,txt http://example.com/documents/

# Reject certain files
wget -r --reject=gif,jpg,png http://example.com/

Authentication

Basic HTTP Authentication

wget --user=username --password=password http://example.com/file.zip

Or prompt for password:

wget --user=username --ask-password http://example.com/file.zip

FTP Authentication

wget ftp://username:password@ftp.example.com/file.zip
wget --ftp-user=username --ftp-password=password ftp://ftp.example.com/file.zip

Read Password from File

echo "mypassword" > ~/.wget-password
chmod 600 ~/.wget-password
wget --password=$(cat ~/.wget-password) --user=username http://example.com/file.zip

Headers and Cookies

Set User-Agent

wget --user-agent="Mozilla/5.0" http://example.com/file.zip

Add Custom Header

wget --header="Authorization: Bearer TOKEN" http://api.example.com/data
wget --header="Accept: application/json" http://api.example.com/data

Save Cookies

wget --save-cookies cookies.txt --keep-session-cookies http://example.com/login

Load Cookies

wget --load-cookies cookies.txt http://example.com/protected-page

# Login and save session
wget --save-cookies cookies.txt \
     --keep-session-cookies \
     --post-data='user=admin&password=secret' \
     http://example.com/login

# Use session to download protected content
wget --load-cookies cookies.txt http://example.com/protected/file.zip

POST Requests

Send POST Data

wget --post-data="param1=value1&param2=value2" http://example.com/api

POST from File

wget --post-file=data.txt http://example.com/api

POST JSON

wget --header="Content-Type: application/json" \
     --post-data='{"key":"value"}' \
     http://api.example.com/endpoint

File Filtering

Accept Only Specific Types

wget -r -A pdf,docx http://example.com/documents/

Reject Specific Types

wget -r -R gif,jpg,jpeg,png http://example.com/

Include/Exclude Directories

# Include specific directories
wget -r -I /directory1,/directory2 http://example.com/

# Exclude specific directories
wget -r -X /ads,/images http://example.com/

Exclude Domains

wget -r --exclude-domains=ads.example.com,tracking.example.com http://example.com/

Follow Only Relative Links

wget -r --no-parent http://example.com/directory/

SSL/TLS Options

Ignore SSL Certificate Errors

wget --no-check-certificate https://self-signed.example.com/file.zip

Specify CA Certificate

wget --ca-certificate=ca-cert.pem https://example.com/file.zip

Use Specific TLS Version

wget --secure-protocol=TLSv1_2 https://example.com/file.zip

Proxy Settings

Use HTTP Proxy

wget --proxy=on --proxy-server=proxy.example.com:8080 http://example.com/file.zip

Proxy with Authentication

wget --proxy-user=user --proxy-password=pass http://example.com/file.zip

No Proxy for Certain Domains

wget --no-proxy http://internal.example.com/file.zip

Advanced Options

Timestamping (Download if Newer)

wget -N http://example.com/file.zip

Only downloads if the remote file is newer than local.

Spider Mode (Check URLs)

wget --spider http://example.com/file.zip

Checks if file exists without downloading.

Server Response Only

wget --server-response http://example.com/file.zip

Debug Mode

wget --debug http://example.com/file.zip

Create Directory Structure

wget -r -nd http://example.com/  # No directories
wget -r -x http://example.com/   # Force directories

Avoid Robot Exclusion

wget -e robots=off http://example.com/

Caution: Only use this if you have permission!

Practical Examples

Download Entire Website for Offline Viewing

wget --mirror \
     --convert-links \
     --adjust-extension \
     --page-requisites \
     --no-parent \
     --wait=1 \
     --random-wait \
     -P /path/to/save \
     http://example.com

Download All Images from Page

wget -r -l 1 -A jpg,jpeg,png,gif http://example.com/gallery/

Download FTP Directory

wget -r ftp://ftp.example.com/pub/files/

Continue Download After Interruption

wget -c http://example.com/large-file.iso

Download with Retry and Wait

wget --retry-connrefused \
     --waitretry=1 \
     --read-timeout=20 \
     --timeout=15 \
     -t 0 \
     http://unreliable-server.com/file.zip

Download from Password-Protected Site

wget --user=admin \
     --ask-password \
     --auth-no-challenge \
     http://protected.example.com/data.zip

Download List with Rate Limit

wget -i urls.txt --wait=2 --limit-rate=100k

Mirror with Subdirectories Only

wget -r -np -nH --cut-dirs=2 http://example.com/path/to/files/

-np: No parent
-nH: No hostname folder
--cut-dirs=2: Skip first 2 directory levels

Download and Rename Pattern

# Download sequentially numbered files
wget http://example.com/file{1..100}.jpg

# With padding
wget http://example.com/file{001..100}.jpg

Background Download with Log

wget -b --progress=bar -o download.log http://example.com/large-file.iso
tail -f download.log

Configuration File

~/.wgetrc

# Default options
timestamping = on
no_clobber = on
timeout = 30
tries = 20
retry_connrefused = on

# Download behavior
continue = on
progress = bar

# Recursive options
recursive = off
level = 0

# User agent
user_agent = Mozilla/5.0 (compatible; MyBot/1.0)

# Proxy (if needed)
# http_proxy = http://proxy.example.com:8080
# use_proxy = on

Comparison: wget vs curl

Feature	wget	curl
Recursive download	✓	✗
Resume download	✓	✓
Multiple protocols	Limited	Extensive
Upload files	✗	✓
Library support	✗	✓ (libcurl)
Background download	✓	✗
Mirror websites	✓	✗
REST API testing	Limited	✓
Default behavior	Save to file	Output to stdout

Use wget for: Downloading files, mirroring websites, recursive downloads
Use curl for: API testing, uploads, protocol flexibility

Exit Status

Code	Meaning
0	No problems occurred
1	Generic error code
2	Parse error (command line options)
3	File I/O error
4	Network failure
5	SSL verification failure
6	Username/password authentication failure
7	Protocol errors
8	Server issued an error response

Tips and Best Practices

Always use -c for large downloads to enable resume
Be respectful - Use --wait and --limit-rate when mirroring
Use -N for synchronization to avoid re-downloading unchanged files
Background downloads - Use -b for long-running downloads
Check with --spider before downloading large files
Use -i for batch downloads from a file list
Mirror responsibly - Add delays with --wait and --random-wait
Respect robots.txt - Only use -e robots=off with permission
Use authentication carefully - Don't put passwords in command history
Monitor progress with --show-progress for single files

Common Options Summary

Option	Description
`-O file`	Save as different filename
`-P dir`	Save to directory
`-c`	Continue/resume download
`-b`	Background download
`-q`	Quiet mode
`-i file`	Read URLs from file
`-r`	Recursive download
`-l depth`	Maximum recursion depth
`-A list`	Accept file types
`-R list`	Reject file types
`-np`	No parent directory
`--mirror`	Mirror website
`-k`	Convert links for offline
`-p`	Download page requisites
`-w seconds`	Wait between retrievals
`--limit-rate`	Limit download speed
`-t number`	Number of retries
`--user`	HTTP/FTP username
`--password`	HTTP/FTP password