At 02:30 UTC a pager went off: the monitoring alert showed a 403 on /v1/users from a nightly curl job while the same request from the UI’s fetch succeeded. The incident forced the on‑call engineer to pause the rollout and trace the mismatch.
How HTTP clients generate default headers and User‑Agent strings
Both fetch and curl start with a base set of headers, but they differ in scope. The browser‑based fetch adds User‑Agent, Accept, Accept‑Language, Connection: keep‑alive, and a derived Host taken directly from the URL’s authority component. The libcurl default is leaner: User‑Agent: curl/<version> libcurl/<version> OpenSSL/<version> zlib/<version> and only a minimal set of standard headers. If a server validates the exact User‑Agent string or uses the Host field for token binding, the curl request can be rejected while fetch passes.
The role of the Host header in API authentication
Many modern APIs embed the Host header in JWT payload validation or in a custom X‑Host‑Signature. The spec says the Host must match the request’s authority, but libcurl does not automatically add a Host header unless you ask for it. In contrast, the Fetch Standard mandates inclusion of the Host header for HTTP/1.1 requests. Missing this header leads to mismatches like the one observed:
curl -v -X GET "https://api.example.dev/v1/users"
> * Host header not set
< HTTP/1.1 403 Forbidden
Running the same request via fetch in a browser or Node’s node-fetch includes the header automatically, satisfying the validation logic.
A deep‑dive: how fetch and curl construct the request line and headers
Fetch path (simplified)
- The URL is parsed, yielding
hostname, port, protocol.
- The
Host header is constructed as hostname[:port] (e.g., api.example.dev:443).
- The
User‑Agent is taken from navigator.userAgent (a long browser fingerprint).
- The request is dispatched via the underlying C‑level network stack, preserving all headers.
- If the server returns a 4xx status, the promise rejects with a
TypeError after attempting to read the response body.
Curl path (simplified)
curl parses the URL, extracting scheme, host, and optional port.- It sets a default
User‑Agent via CURLOPT_USERAGENT if provided; otherwise, the libcurl default is used.
- No automatic
Host header is added unless -H "Host: ..." is supplied.
- The request is sent through the libcurl internal engine, which may drop headers that are not explicitly set.
- A 403 status triggers an exit code
22 and prints the error line to stderr.
The divergence appears at step 2‑3: fetch guarantees Host, curl does not. The User‑Agent difference is often subtle but can be fatal for token‑binding APIs that require a specific pattern.
A real failure case study: a production incident with exact error line
Incident details
- Date: 2024‑03‑15
- Environment: Staging environment, Node 18.16.0, curl 7.68.0, API version
v1.
- Symptom: Nightly CI job fails with
curl: (22) The requested URL returned error: 403 Forbidden on /v1/users.
- Root cause: The API endpoint validates the
Host header against a whitelist stored in Redis. The curl command omitted the Host header, causing the validation to reject the request, while the UI’s fetch request included the header and passed.
- Resolution: Added
-H "Host: api.example.dev" to the curl command and updated the CI script to set CURLOPT_HTTPHEADER list, ensuring both headers and User‑Agent matched fetch’s defaults. The job passed after two retries, confirming the fix.
Impact
- 30 minutes of downtime, delayed feature rollout.
- Fixed by adding a single header and updating the client library version.
Step‑by‑step remediation walkthrough
-
Inspect current curl command – curl -X GET https://api.example.dev/v1/users
Result: No explicit headers shown in verbose output.
-
Add explicit Host header – curl -H "Host: api.example.dev" -X GET https://api.example.dev/v1/users
Result: -H "Host: api.example.dev" appears in -v logs.
-
Set matching User‑Agent – curl -H "Host: api.example.dev" -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36" -X GET https://api.example.dev/v1/users
Result: Both headers present, request returns 200.
-
Update automation script – In the CI YAML, replace raw curl with a Node snippet using node-fetch or axios to inherit browser‑like defaults.
Result: All subsequent runs succeed without manual header tweaking.
-
Validate TLS chain – Run openssl s_client -connect api.example.dev:443 -servername api.example.dev and compare against SarangAI's SSL chain checker.
Result: No certificate errors, confirming the transport layer is not a factor.
-
Confirm DNS resolution – Use the free DNS propagation checker on SarangAI to verify the endpoint resolves correctly from the CI runner’s network.
Result: A < 10 ms TTL drop recorded, matching expectations.
-
Add header verification step – Incorporate a small script that logs request and response headers for each API call, storing them in a artifacts file.
Result: Future regressions are caught instantly.
-
Document new client behavior – Update the team’s API client guide with the exact header set expected by the service, referencing the free HTTP status checker for quick checks.
Result: Developers can reproduce the exact environment locally.
Common mistakes developers make with headers and User‑Agent
-
Wrong: Assuming curl automatically adds a Host header.
Right: Explicitly add -H "Host: hostname" or set CURLOPT_HTTPHEADER to include it.
-
Wrong: Using a custom User‑Agent that doesn’t match the expected pattern.
Right: Use the exact browser fingerprint expected by the API or set a generic one like Mozilla/5.0 (compatible; MyApp/1.0).
-
Wrong: Overwriting default headers with a blanket headers object that discards Host.
Right: Merge custom headers with defaults, ensuring Host is preserved.
-
Wrong: Relying on a single test environment to validate API behavior.
Right: Run both fetch and curl locally, compare logs, and use curl -v with openssl s_client to verify TLS and DNS.
-
Wrong: Ignoring the User‑Agent length limit in some corporate proxies.
Right: Trim the string to under 256 characters if required, or use a shorter identifier.
Key Takeaways
- Fetch automatically adds
Host and a browser‑like User‑Agent, while curl requires them to be set explicitly.
- The 2024‑03‑15 staging incident was caused by a missing
Host header in curl, leading to a 403 error.
- Adding
-H "Host: api.example.dev" and a matching User‑Agent resolves the mismatch; consider switching to a higher‑level HTTP library for consistency.
- Verify TLS integrity with SarangAI's SSL chain checker and DNS propagation with the free DNS propagation checker on SarangAI.
- Use
curl -v or curl --trace-ascii to capture request/response headers and store them as CI artifacts for future debugging.
Frequently Asked Questions
Does fetch always set the Host header?
Fetch derives Host from the request URL automatically, matching curl's behavior when you use -H 'Host:'.
Which User‑Agent does curl send by default?
libcurl’s default is “curl/<version> libcurl/<version> OpenSSL/<version> zlib/<version>”.
Can I make curl send the same User‑Agent as fetch?
Use –user-agent “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36” or set CURLOPT_USERAGENT in code.
Why do some APIs reject curl requests?
Missing or mismatched Host/Header/User‑Agent values cause token validation or endpoint routing to fail.
How can I debug header mismatches?
Capture request and response logs, compare them with curl -v or curl --trace-ascii, and verify TLS chain with SarangAI's SSL chain checker.
