At 02:34 UTC on 12 May 2024 a pager blared in the ops Slack. The incident page listed:
- HTTP 500 from
api.example.dev for a single user’s GET /v1/auth.
- Client‑side console: “Redirected 12 times – maximum reached.”
- Edge logs from the CDN (Fastly) contained identical “penalty box” entries.
The root cause was a CNAME record that still pointed to an old CDN endpoint, while the origin server returned a Location header that included the new CDN domain. The mismatch formed a loop only visible when the production DNS had been updated but the CDN rule had not.
How a CDN Redirect Loop Actually Works
A loop begins at the DNS level. The client resolves api.example.dev to the CDN edge IP. The CDN evaluates its configuration:
- Request enters edge – the HTTP request is matched against the
request.settings.
- Rewrite rule fires – a
bereq.url rewrite changes the path to /old-auth.
- Origin request – the edge forwards to
origin.example.com/old-auth.
- Origin response – the application returns
HTTP 302 Location: https://cdn.example.net/v1/auth.
- CDN rewrite – the
Location header is rewritten again to https://cdn.example.net/old-auth.
- Client receives – the browser follows the new Location, which points back to step 2.
If the CNAME still resolves to the old edge IP, the client will never reach the intended origin. The loop repeats until the client’s redirect counter (default 5) is exceeded, then the CDN returns HTTP 500 with “Too many redirects”. The internal latency spikes to >30 seconds, and the error log shows curl: (30) The requested URL returned error: 301 multiple times.
The penalty‑box logic in Fastly logs each repeated request with a request ID (req.id: abcdef12345678). Seeing the same req.id across multiple penalty_box entries is a strong signal that the CDN is looping. The edge also logs the bereq.ttl and cacheable flags, which can be inspected to confirm that the object is not being cached because the redirect rule never yields a 200.
Production Incident: 2 a.m. Pager Alert
The on‑call engineer opened the incident and saw the timeline:
- 00:00 – DNS change pushed to production (CNAME updated to
cdn.example.net).
- 00:05 – CDN cache flush completed.
- 02:30 – First user reported auth failure.
- 02:34 – Pager triggered.
The engineer used dig +short api.example.dev and observed 23.45.12.89 (old edge). The new CNAME pointed to 98.76.54.32. The mismatch explained why the loop only appeared after the DNS update. The incident was resolved after a manual purge of the old CNAME from the DNS manager and a verification that the CDN’s origin field matched the new origin server.
The engineer also checked the Fastly service version history and discovered a stale request_settings rule that forced http_to_https on the new domain. That rule was rewriting the Location header incorrectly. A quick dig api.example.dev confirmed that global vantage points were still seeing the old IP, while the internal DNS resolver saw the new one. A curl -v https://cdn.example.net/v1/auth request printed the looping Location headers in real time, and the free redirect checker confirmed the chain, making the loop obvious even before the CDN cache purge.
The total time from first symptom to resolution was 45 minutes. The post‑mortem documented the exact error line from the origin: Location: https://cdn.example.net/v1/auth, which was generated by an outdated version of the authentication service still deployed in production.
Step‑by‑Step Remediation (7 steps)
- Flush the CDN cache –
fastly service purge -s <service_id>. Expected: all edge objects expire within 30 seconds.
- Verify CNAME propagation –
dig api.example.dev. Expected: resolves to the new edge IP (98.76.54.32).
- Check origin configuration – In the CDN control panel, confirm the origin URL is
origin.example.com.
- Inspect rewrite rules –
curl -I https://cdn.example.net/old-auth. Expected: HTTP/2 200 or a proper redirect, not a loop.
- Disable redirect rules temporarily – Set
override_origin to false. Expected: request reaches origin directly.
- Test the flow – Compare DNS and HTTP with standard tools:
dig +trace api.example.dev && curl -v https://cdn.example.net/v1/auth. Expected: single redirect to origin and success.
- Re‑enable rules and purge again –
fastly service version activate -v <new_version>. Expected: stable traffic with no loops.
Each step includes a concrete command and a clear “expected result” to ensure the engineer does not skip validation. Add a unique X-Debug-ID header to each curl request so it can be correlated with edge logs during remediation.
Common Mistakes When Debugging CDN Redirects
-
Wrong origin URL – Using https://cdn.example.net as origin creates a self‑reference loop.
Correct: Use the raw origin server address (origin.example.com).
-
Stale CNAME TTL – Leaving a 24‑hour TTL on the old CDN record keeps the old IP in caches.
Correct: Set TTL to 300 seconds for rapid rollouts.
-
Mis‑configured Location header – The application’s Location includes the CDN domain again, causing double rewrite.
Correct: Ensure Location points only to the origin or a relative path.
-
Cache key mismatch – Different Accept‑Encoding values cause separate cache entries that bypass the intended redirect.
Correct: Align cache_key settings with the redirect rule conditions.
-
Missing X‑Forwarded‑Proto – The CDN thinks the request is HTTP and forces HTTP redirects, while the origin expects HTTPS.
Correct: Enable force_tls or set the header appropriately.
These pitfalls are frequently seen in production after a CDN migration. A quick dig can reveal an unexpected CNAME, while the SarangAI SSL chain checker validates that certificate changes are not introducing unintended redirects.
Real‑World Debugging Tools
You can use the free DNS propagation checker on SarangAI to confirm that the CNAME record points correctly across global vantage points. The SarangAI's SSL chain checker helps you verify that certificate chaining does not introduce unexpected redirects. For a quick command‑line audit, use dig to inspect DNS and curl -v to log every redirect hop:
dig +trace api.example.dev
curl -v -L --max-redirs 5 -H "X-Debug-ID: $(date +%s)" https://cdn.example.net/v1/auth
To check the network path itself, use traceroute or mtr against the CDN hostname. The -v flag on curl prints the raw request and response headers, making a duplicated Location header easy to spot.
Code Examples: Detecting Loops with curl and Python
# cURL session that reproduces the loop
curl -v -L --max-time 5 --location "https://cdn.example.net/v1/auth" 2>&1 | head -n 30
Typical output:
> GET /v1/auth HTTP/1.1
> Host: cdn.example.net
< HTTP/2 301
< Location: https://cdn.example.net/old-auth
> GET /old-auth HTTP/1.1
< HTTP/2 301
< Location: https://cdn.example.net/v1/auth
...
curl: (30) The requested URL returned error: 301 multiple times
import requests
from urllib.parse import urlparse
def follow_redirects(url, max_hops=10):
visited = set()
for _ in range(max_hops):
resp = requests.get(url, allow_redirects=False)
if resp.status_code not in (301, 302, 303, 307, 308):
return resp
location = resp.headers.get('Location')
if not location:
break
parsed = urlparse(location)
# Resolve relative URLs
if not parsed.netloc:
base = urlparse(url)
location = f"{base.scheme}://{base.netloc}{parsed.path}"
if location in visited:
raise Exception(f"Redirect loop detected at {location}")
visited.add(location)
url = location
raise Exception(f"Exceeded max redirect hops ({max_hops})")
try:
resp = follow_redirects("https://cdn.example.net/v1/auth")
print(f"Final status: {resp.status_code}")
except Exception as e:
print(f"Loop found: {e}")
The script prints a clear error once it revisits a URL, making it easy to integrate into CI pipelines.
Key Takeaways
- A loop appears when CNAME TTL, CDN origin, and application Location headers are out of sync.
- Flushing CDN cache and confirming DNS propagation are the fastest ways to isolate the problem.
- The free DNS propagation checker on SarangAI verifies global CNAME consistency in seconds.
- Mis‑configured origin URLs cause the most common self‑reference loops.
- Using
curl -v -L and a Python redirect‑tracker provides both visual and programmatic proof of loops.
Frequently Asked Questions
What are the most common causes of a CDN redirect loop?
Mis‑configured origin URL, stale CNAME TTL, and application Location headers that include the CDN domain again. Each of these creates a circular path between edge and origin.
How does the loop manifest in client logs?
Clients see repeated HTTP 301/302 responses, increasing latency, and finally a “Redirected X times – maximum reached” error before a 500 is returned by the CDN.
Which command quickly confirms a loop?
curl -v -L --max-time 5 https://cdn.example.net/v1/auth will output the series of Location headers and terminate with “curl: (30) The requested URL returned error: 301 multiple times” when a loop exists.
Can standard CLI tools help diagnose the issue?
Yes, dig api.example.dev shows the current edge IP, and curl -v https://cdn.example.net/v1/auth logs each redirect step; add an X-Debug-ID header to correlate requests in edge logs.
Why does the loop only appear in production?
Production DNS often has longer TTL values and different cache rules than staging, exposing mis‑alignments that are hidden in lower‑environment configurations.