wget Command

What is Linux wget Command?#

wget is a free GNU command-line utility for downloading files from the web. The name stands for “World Wide Web” and “get”. It retrieves files using HTTP, HTTPS, and FTP protocols and is designed to work robustly in slow or unstable network connections. wget is non-interactive, meaning it can work in the background while you’re not logged in. This makes it perfect for downloading large files, mirroring websites, and automating download tasks through scripts. From the man page: > GNU Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies. ## Key Features

• Protocol support: HTTP, HTTPS, and FTP downloads- Resume capability: Continue interrupted downloads - Recursive downloading: Download entire websites - Rate limiting: Control download speed - Authentication: Support for various authentication methods - Proxy support: Work through HTTP and HTTPS proxies - Background operation: Run downloads in the background - Robust networking: Handle unstable connections gracefully ## wget Syntax wget [options] [URL] Basic usage patterns: - wget URL - Download single file - wget -O filename URL - Save with custom name - wget -c URL - Resume interrupted download - wget -r URL - Recursive download ## Installation ### Most Linux Distributions wget comes pre-installed on most Linux distributions. If not available: # Ubuntu/Debian sudo apt update && sudo apt install wget # CentOS/RHEL/Fedora sudo yum install wget # CentOS/RHEL sudo dnf install wget # Fedora # Arch Linux sudo pacman -S wget ### macOS # Using Homebrew brew install wget # Using MacPorts sudo port install wget ### Windows Download from GNU Wget official website or use Windows Subsystem for Linux (WSL). ## Basic wget Examples ### Simple File Download Download a single file: wget https://example.com/file.zip ### Download with Custom Filename Save file with a different name: wget -O myfile.zip https://example.com/file.zip ### Download to Specific Directory Download file to a specific location: wget -P /downloads/ https://example.com/file.zip ### Download Multiple Files Download multiple files by listing URLs: wget https://example.com/file1.zip https://example.com/file2.zip Download from URL list in file: wget -i urls.txt ## Essential wget Options ### Resume Interrupted Downloads (-c) Continue a partially downloaded file: wget -c https://example.com/largefile.iso ### Background Downloads (-b) Run download in background: wget -b https://example.com/largefile.zip Check background download progress: tail -f wget-log ### Quiet Mode (-q) Suppress output (useful for scripts): wget -q https://example.com/file.zip ### Verbose Mode (-v) Show detailed output: wget -v https://example.com/file.zip ### Limit Download Rate (–limit-rate) Limit bandwidth usage: wget --limit-rate=200k https://example.com/file.zip # 200 KB/s wget --limit-rate=1m https://example.com/file.zip # 1 MB/s ## Advanced wget Features ### Authentication #### Basic HTTP Authentication wget --user=username --password=password https://secure.example.com/file.zip #### Certificate-based Authentication wget --certificate=client-cert.pem --private-key=private-key.pem https://secure.example.com/ ### Working with Cookies Save cookies to file: wget --save-cookies cookies.txt https://example.com/login Load cookies from file: wget --load-cookies cookies.txt https://example.com/protected-area Keep session cookies: wget --keep-session-cookies --save-cookies cookies.txt https://example.com/ ### Custom Headers Add custom HTTP headers: wget --header="User-Agent: MyBot/1.0" https://example.com/api/data wget --header="Authorization: Bearer token123" https://api.example.com/data ### Retry Options Set retry attempts and delays: wget --tries=5 --retry-connrefused --waitretry=10 https://unreliable-server.com/file.zip ## Recursive Downloading ### Download Entire Website Basic website mirror: wget -r https://example.com/ ### Advanced Website Mirroring Complete website download with optimizations: wget -r -k -l 7 -p -E -nc -np https://example.com/ Options explained: - -r: Recursive download - -k: Convert links for local viewing - -l 7: Limit recursion depth to 7 levels - -p: Download page requisites (images, CSS, etc.) - -E: Add .html extension to files - -nc: No clobber (don’t overwrite existing files) - -np: No parent (don’t go up in directory structure) ### Selective Website Download Download only specific file types: wget -r -A "*.pdf,*.doc,*.docx" https://example.com/documents/ Exclude specific file types: wget -r -R "*.gif,*.jpg,*.jpeg,*.png" https://example.com/ ### Mirror with Time Delays Add delays between requests (be respectful to servers): wget -r -w 2 --random-wait https://example.com/ ## Working with FTP ### Basic FTP Download wget ftp://ftp.example.com/pub/file.tar.gz ### FTP with Authentication wget --ftp-user=username --ftp-password=password ftp://ftp.example.com/private/file.zip ### FTP Directory Listing wget -r -l1 --no-remove-listing ftp://ftp.example.com/pub/ ## Practical Use Cases ### System Updates and Software Download Linux ISO: wget -c https://releases.ubuntu.com/22.04/ubuntu-22.04-desktop-amd64.iso Download and install software: wget https://golang.org/dl/go1.19.linux-amd64.tar.gz sudo tar -C /usr/local -xzf go1.19.linux-amd64.tar.gz ### Backup and Archiving Create website backup: wget -r -k -p -E -np -nc --convert-links --adjust-extension https://mywebsite.com/ Download git repositories: wget https://github.com/user/repo/archive/main.zip ### API Data Collection Download JSON data: wget -O data.json "https://api.example.com/data?format=json" Download with API key: wget --header="X-API-Key: your-api-key" https://api.example.com/data ### Automation Scripts Daily backup script: #!/bin/bash DATE=$(date +%Y%m%d) wget -r -k -p -E -nc https://example.com/ -P /backups/$DATE/ Monitor file changes: #!/bin/bash wget -N https://example.com/data.xml # Download only if newer ## Troubleshooting and Debugging ### SSL/TLS Issues Ignore SSL certificate errors (not recommended for production): wget --no-check-certificate https://self-signed.example.com/file.zip Specify CA certificate: wget --ca-certificate=ca-cert.pem https://secure.example.com/ ### Connection Problems Increase timeout values: wget --timeout=60 --tries=3 https://slow-server.com/file.zip Use different user agent (some sites block wget): wget --user-agent="Mozilla/5.0 (compatible; MyBrowser/1.0)" https://example.com/ ### Debugging Network Issues Enable debug output: wget --debug https://example.com/file.zip Show response headers: wget -S https://example.com/file.zip ### Handling Redirects Follow redirects automatically: wget --max-redirect=5 https://bit.ly/short-url ### Large File Downloads For very large files, use these optimizations: wget -c -t 0 --timeout=60 --waitretry=60 https://example.com/huge-file.tar.gz ## wget Configuration ### Configuration File Create ~/.wgetrc for default settings: # Default options tries = 3 timeout = 60 continue = on progress = bar # User agent user_agent = Mozilla/5.0 (compatible; wget/1.21) # Rate limiting limit_rate = 500k # Directories dir_prefix = /home/user/downloads ### Environment Variables Set default options via environment: export WGETRC=~/.wgetrc export https_proxy=http://proxy.example.com:8080 ## Performance Optimization ### Parallel Downloads Download multiple files simultaneously: wget URL1 & wget URL2 & wget URL3 & wait ### Optimize for Speed Fast download with minimal overhead: wget --progress=dot:mega --no-dns-cache https://example.com/file.zip ### Server-Friendly Practices Respectful website mirroring: wget -r -w 1 --random-wait --user-agent="Friendly Bot 1.0"- \--reject="*.gif,*.jpg,*.jpeg,*.png" \ \--exclude-directories="/images,/media" \ https://example.com/ ## Security Considerations ### Safe Download Practices 1. Verify checksums: wget https://example.com/file.zip wget https://example.com/file.zip.sha256 sha256sum -c file.zip.sha256 2. Use HTTPS when available: wget https://secure.example.com/file.zip # Preferred wget http://insecure.example.com/file.zip # Avoid if possible 3. Validate certificates: wget --ca-directory=/etc/ssl/certs https://secure.example.com/ ### Secure Authentication Store credentials securely: # Use .netrc file for credentials echo "machine example.com login user password pass" >> ~/.netrc chmod 600 ~/.netrc wget https://example.com/protected/file.zip ## wget vs Other Tools ### wget vs curl - wget: Recursive downloads, website mirroring, simpler for basic downloads - curl: More protocols, better API testing, more flexible for complex requests ### wget vs browsers - wget: Scriptable, resumable, efficient for large files - Browsers: Interactive, JavaScript support, better for dynamic content ### wget vs rsync - wget: Web downloads, one-way transfers - rsync: Bidirectional sync, incremental transfers, better for existing file collections ## Error Handling and Scripting ### Basic Error Handling #!/bin/bash if wget -q https://example.com/file.zip; then echo "Download successful" else echo "Download failed with exit code $?" exit 1 fi ### Advanced Download Script #!/bin/bash URL="https://example.com/file.zip" OUTPUT="/downloads/file.zip" MAX_ATTEMPTS=3 for i in $(seq 1 $MAX_ATTEMPTS); do echo "Attempt $i of $MAX_ATTEMPTS" if wget -c -O "$OUTPUT" "$URL"; then echo "Download successful" exit 0 else echo "Attempt $i failed" if [ $i -eq $MAX_ATTEMPTS ]; then echo "All attempts failed" exit 1 fi sleep 10 fi done ## wget Command Manual / Help We can use man and info command to see the manual page of wget command. wget command also have --help option to show list of options. To open man page for wget command we can use command below. To exit man or info page you can press q. man wget To open info page for wget command we can use command below. info wget To open help page from wget command we can run command below. wget --help Get version information: wget --version ## References You can find more information about wget from the following links:

• GNU Wget Official Website

• wget Project Page

• wget Manual

• wget Mirror on GitHub

Related Linux Commands#

You can read tutorials of related Linux commands below:

• curl Command- rsync Command * scp Command ## Summary In this comprehensive tutorial, we’ve covered the essential aspects of using wget for downloading files from the web. wget is a powerful and reliable tool for automated downloads, website mirroring, and file retrieval tasks. Key takeaways:- wget excels at downloading files over HTTP, HTTPS, and FTP protocols - Resume capability makes it perfect for large files and unstable connections - Recursive downloading enables complete website mirroring - Extensive options provide fine control over download behavior - Built-in retry mechanisms handle network issues gracefully - Configuration files and scripting enable automation - Essential tool for system administrators and power users Visit our Linux Commands guide to learn more about using command line interface in Linux.