curl Mastery

Overview

curl is the Swiss Army knife of HTTP. Master these patterns for API debugging, automation, security testing, and data extraction.

Essential Options

Option Purpose Example

-s

Silent (no progress)

curl -s api.example.com

-S

Show errors with silent

curl -sS api.example.com

-f

Fail silently on HTTP errors

curl -sf api.example.com

-L

Follow redirects

curl -L short.url/abc

-o

Output to file

curl -o file.json api.example.com

-O

Save with remote filename

curl -O example.com/file.tar.gz

-I

Headers only (HEAD request)

curl -sI example.com

-i

Include headers in output

curl -si api.example.com

-v

Verbose (debug)

curl -v api.example.com

-k

Skip TLS verification

curl -k self-signed.local

-w

Custom output format

curl -w '%\{http_code}' -o /dev/null

HTTP Methods

# GET (default)
curl https://api.example.com/users

# POST with JSON
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{"name": "John", "email": "john@example.com"}'

# PUT
curl -X PUT https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"name": "John Updated"}'

# PATCH
curl -X PATCH https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"email": "new@example.com"}'

# DELETE
curl -X DELETE https://api.example.com/users/123

# HEAD (check if resource exists)
curl -I https://api.example.com/users/123

Authentication

Bearer Token (API)

curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com/data

Basic Auth

# Inline (avoid - visible in history)
curl -u username:password https://api.example.com

# Prompt for password
curl -u username https://api.example.com

# From variable (better)
curl -u "$USER:$PASS" https://api.example.com

# Netrc file (best for automation)
curl -n https://api.example.com
# Requires ~/.netrc with: machine api.example.com login USER password PASS

Client Certificates (mTLS)

curl --cert /path/to/client.crt \
     --key /path/to/client.key \
     --cacert /path/to/ca.crt \
     https://mtls.example.com

Cloudflare Access Service Token

curl -H "CF-Access-Client-Id: $CF_ACCESS_CLIENT_ID" \
     -H "CF-Access-Client-Secret: $CF_ACCESS_CLIENT_SECRET" \
     https://protected.example.com

Headers

# Single header
curl -H "Accept: application/json" https://api.example.com

# Multiple headers
curl -H "Accept: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -H "X-Request-ID: $(uuidgen)" \
     https://api.example.com

# Custom User-Agent
curl -A "MyApp/1.0" https://api.example.com

# Referer
curl -e "https://google.com" https://example.com

Data Sending

JSON Body

# Inline JSON
curl -X POST https://api.example.com/data \
  -H "Content-Type: application/json" \
  -d '{"key": "value"}'

# From file
curl -X POST https://api.example.com/data \
  -H "Content-Type: application/json" \
  -d @payload.json

# From stdin
echo '{"key": "value"}' | curl -X POST https://api.example.com/data \
  -H "Content-Type: application/json" \
  -d @-

Form Data

# URL-encoded form
curl -X POST https://example.com/login \
  -d "username=admin&password=secret"

# Multipart form (file upload)
curl -X POST https://example.com/upload \
  -F "file=@/path/to/file.pdf" \
  -F "description=My document"

# Multiple files
curl -X POST https://example.com/upload \
  -F "files[]=@file1.jpg" \
  -F "files[]=@file2.jpg"

Response Processing

Status Code Only

curl -s -o /dev/null -w '%\\{http_code}' https://example.com

Headers + Body

curl -si https://api.example.com | head -20

Extract Specific Header

curl -sI https://example.com | awk -F': ' '/^content-type:/{print $2}'

JSON Response with jq

# Pretty print
curl -s https://api.example.com/users | jq

# Extract field
curl -s https://api.example.com/users | jq '.[0].name'

# Filter array
curl -s https://api.example.com/users | jq '.[] | select(.active == true)'

# Transform to CSV
curl -s https://api.example.com/users | jq -r '.[] | [.id, .name, .email] | @csv'

Conditional on Status

response=$(curl -s -w "\n%\\{http_code}" https://api.example.com)
body=$(echo "$response" | head -n -1)
status=$(echo "$response" | tail -n 1)

if [[ "$status" == "200" ]]; then
    echo "$body" | jq
else
    echo "Error: $status"
fi

Timing and Performance

Detailed Timing

curl -w @- -o /dev/null -s https://example.com << 'EOF'
    time_namelookup:  %{time_namelookup}s
    time_connect:     %{time_connect}s
    time_appconnect:  %{time_appconnect}s
    time_pretransfer: %{time_pretransfer}s
    time_redirect:    %{time_redirect}s
    time_starttransfer: %{time_starttransfer}s
    ----------
    time_total:       %{time_total}s
EOF

Quick Latency Check

curl -s -o /dev/null -w '%{time_total}s\n' https://example.com

TTFB (Time to First Byte)

curl -s -o /dev/null -w '%{time_starttransfer}s\n' https://example.com

TLS/SSL Inspection

View Certificate Chain

curl -vI https://example.com 2>&1 | awk '/SSL connection|subject:|issuer:|expire/'

Certificate Expiry

curl -vI https://example.com 2>&1 | grep "expire date"

Test Specific TLS Version

# TLS 1.2 only
curl --tlsv1.2 --tls-max 1.2 https://example.com

# TLS 1.3 only
curl --tlsv1.3 https://example.com

View Cipher Used

curl -vI https://example.com 2>&1 | grep "SSL connection using"

Debugging

Full Debug Output

curl -v https://api.example.com 2>&1

Trace Everything (hex dump)

curl --trace - https://api.example.com

Trace to File

curl --trace-ascii /tmp/curl_trace.txt https://api.example.com

Show Request Headers Only

curl -v https://api.example.com 2>&1 | grep "^>"

Show Response Headers Only

curl -v https://api.example.com 2>&1 | grep "^<"

API Patterns

Cloudflare API

# Verify token
curl -s -H "Authorization: Bearer $CF_API_TOKEN" \
  https://api.cloudflare.com/client/v4/user/tokens/verify | jq

# List zones
curl -s -H "Authorization: Bearer $CF_API_TOKEN" \
  https://api.cloudflare.com/client/v4/zones | jq '.result[] | {name, id}'

# Create DNS record
curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"type":"A","name":"test","content":"10.0.0.1","ttl":1}' | jq

GitHub API

# User info
curl -s -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/user | jq '{login, name, email}'

# List repos
curl -s -H "Authorization: token $GITHUB_TOKEN" \
  "https://api.github.com/user/repos?per_page=100" | jq '.[].full_name'

# Create repo
curl -s -X POST -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/user/repos \
  -d '{"name":"new-repo","private":true}' | jq

GitLab API

# List projects
curl -s -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.com/api/v4/projects?owned=true" | jq '.[].path_with_namespace'

# Create project
curl -s -X POST -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.com/api/v4/projects" \
  -d "name=new-project&visibility=private" | jq

Vault API

# Read secret
curl -s -H "X-Vault-Token: $VAULT_TOKEN" \
  "$VAULT_ADDR/v1/secret/data/myapp" | jq '.data.data'

# Write secret
curl -s -X POST -H "X-Vault-Token: $VAULT_TOKEN" \
  "$VAULT_ADDR/v1/secret/data/myapp" \
  -d '{"data": {"username": "admin", "password": "secret"}}' | jq

Download Patterns

Resume Interrupted Download

curl -C - -O https://example.com/large-file.iso

Download with Progress

curl -# -O https://example.com/file.tar.gz

Limit Bandwidth

curl --limit-rate 1M -O https://example.com/file.iso

Download Multiple Files

curl -O https://example.com/file1.txt -O https://example.com/file2.txt

# From file list
xargs -n1 curl -O < urls.txt

Loops and Automation

Retry on Failure

curl --retry 3 --retry-delay 2 --retry-connrefused https://api.example.com

Pagination

page=1
while true; do
    response=$(curl -s "https://api.example.com/items?page=$page&per_page=100")
    count=$(echo "$response" | jq 'length')
    [[ "$count" == "0" ]] && break
    echo "$response" | jq -r '.[].name'
    ((page++))
done

Parallel Requests

# Using xargs
cat urls.txt | xargs -P 10 -I {} curl -s {}

# Using GNU parallel
parallel -j 10 curl -s {} ::: $(cat urls.txt)

Poll Until Condition

while true; do
    status=$(curl -s https://api.example.com/job/123 | jq -r '.status')
    [[ "$status" == "complete" ]] && break
    [[ "$status" == "failed" ]] && { echo "Failed"; exit 1; }
    echo "Status: $status"
    sleep 5
done

Security Testing

Check Security Headers

curl -sI https://example.com | grep -iE '^(strict-transport|x-frame|x-content-type|content-security|x-xss)'

Test CORS

curl -sI -H "Origin: https://attacker.com" https://api.example.com | grep -i access-control

Check for Open Redirect

curl -sI "https://example.com/redirect?url=https://evil.com" | grep -i location

Test Rate Limiting

for i in {1..50}; do
    code=$(curl -s -o /dev/null -w '%\\{http_code}' https://api.example.com/endpoint)
    echo "$i: $code"
    [[ "$code" == "429" ]] && { echo "Rate limited at $i"; break; }
done

Proxy and Network

HTTP Proxy

curl -x http://proxy:8080 https://example.com

# With auth
curl -x http://user:pass@proxy:8080 https://example.com

SOCKS5 Proxy

curl --socks5 localhost:1080 https://example.com

# With DNS through proxy
curl --socks5-hostname localhost:1080 https://example.com

Resolve Hostname to IP (bypass DNS)

curl --resolve example.com:443:10.0.0.1 https://example.com

Use Specific Interface

curl --interface eth0 https://example.com

One-Liners

Get Public IP

curl -s ifconfig.me
curl -s icanhazip.com
curl -s ipinfo.io/ip

Weather

curl wttr.in/NewYork
curl -s "wttr.in/NewYork?format=3"

Dictionary

curl dict://dict.org/d:ubiquitous

Check Site Down for Everyone

curl -s "https://downforeveryoneorjustme.com/example.com" | grep -o "It's just you\|It's not just you"

QR Code

curl "qrcode.show/https://example.com"

Config File

Create ~/.curlrc for persistent defaults:

# Progress bar
progress-bar

# Follow redirects
location

# Silent mode
silent

# Show errors
show-error

# User agent
user-agent = "MyCLI/1.0"

# Default headers
header = "Accept: application/json"

Troubleshooting

Connection Refused

# Check if port is open
curl -v telnet://hostname:port

SSL Certificate Problem

# Skip verification (dev only)
curl -k https://self-signed.local

# Specify CA bundle
curl --cacert /path/to/ca-bundle.crt https://example.com

Timeout

# Connection timeout (seconds)
curl --connect-timeout 5 https://slow-server.com

# Max time for entire operation
curl -m 30 https://slow-server.com

Related