Global CDN Architecture Deep Dive
July 23, 2026113 min read0 views
Global CDN Architecture Deep Dive
Introduction
A Content Delivery Network isn't just a cache in front of your origin. At scale, CDN architecture determines whether your global user base experiences sub-100ms loads or multi-second delays. The difference between a well-architected CDN strategy and a naive "just put CloudFront in front of it" approach can be 10x in latency and 100x in origin load.
This deep dive examines CDN architecture from the perspective of a frontend engineer who needs to understand: how requests flow through the network, where caching actually happens, why cache invalidation is genuinely hard, and how to design cache hierarchies that scale.
We'll go beyond marketing materials and examine the actual mechanics: anycast routing, PoP internals, cache key design, tiered caching, and the coordination problems that make global cache invalidation surprisingly complex.
Scale Context
Production CDN deployments we're examining:
| Metric | Value |
|---|---|
| Global PoPs | 200-300 |
| Requests per Second | 10M+ |
| Cache Hit Ratio (target) | 95%+ |
| Origin Requests per Second | <500K (5% miss rate) |
| Cached Objects | Billions |
| Total Cache Storage | Petabytes |
| Geographic Coverage | 6 continents, 100+ countries |
| P50 Edge Latency | <20ms |
| P99 Edge Latency | <100ms |
| Origin Shield Regions | 3-5 |
| Cache Invalidation Time (global) | <60 seconds |
At this scale, a 1% improvement in cache hit ratio reduces origin load by 200K requests per second.
Network Topology
Anycast and GeoDNS
CDNs use two primary mechanisms to route users to nearby PoPs:
Anycast: Multiple PoPs announce the same IP address via BGP. Routers naturally select the "closest" path based on BGP metrics.
GeoDNS: DNS servers return different IPs based on the requester's geographic location.
┌─────────────────────────────────────────────────────────────────────────────┐
│ ANYCAST ROUTING │
│ │
│ User in Tokyo requests 104.16.123.45 │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ BGP ROUTING TABLE │ │
│ │ │ │
│ │ Network: 104.16.0.0/16 │ │
│ │ │ │
│ │ Announcements: │ │
│ │ ├── Tokyo PoP: AS Path: 13335 (1 hop) │ │
│ │ ├── Singapore PoP: AS Path: 13335, 7473 (2 hops) │ │
│ │ ├── San Jose PoP: AS Path: 13335, 3356, 2914 (3 hops) │ │
│ │ │ │
│ │ Router Decision: Tokyo (shortest AS path) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Anycast Advantages: │
│ • Automatic failover (PoP goes down → BGP withdraws → traffic reroutes) │
│ • No DNS TTL delays during failover │
│ • Single IP simplifies DNS and TLS certificates │
│ │
│ Anycast Limitations: │
│ • BGP metrics ≠ latency (AS path length is a proxy, not direct measure) │
│ • Long TCP connections may break during BGP changes │
│ • Limited control over routing decisions │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ GEODNS ROUTING │
│ │
│ User in Tokyo requests cdn.example.com │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ DNS RESOLUTION │ │
│ │ │ │
│ │ Authoritative Server receives query from resolver │ │
│ │ ├── Source IP: 8.8.8.8 (Google DNS) │ │
│ │ ├── EDNS Client Subnet: 203.0.113.0/24 (user's /24) │ │
│ │ │ │
│ │ GeoIP Lookup: 203.0.113.0 → Tokyo, Japan │ │
│ │ │ │
│ │ Response: cdn.example.com → 198.51.100.1 (Tokyo PoP) │ │
│ │ TTL: 60 seconds │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ GeoDNS Advantages: │
│ • Direct control over routing decisions │
│ • Can route based on actual latency measurements │
│ • Supports weighted routing for load balancing │
│ │
│ GeoDNS Limitations: │
│ • Failover limited by DNS TTL (users cache stale IPs) │
│ • EDNS Client Subnet not universally supported │
│ • Resolver location may not match user location │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
PoP Internal Architecture
A Point of Presence (PoP) contains multiple layers of infrastructure:
┌─────────────────────────────────────────────────────────────────────────────┐
│ PoP ARCHITECTURE │
│ │
│ Incoming Traffic ──────────────────────────────────────────────────────▶ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ NETWORK LAYER │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Border │ │ Border │ │ Border │ │ │
│ │ │ Router │──│ Router │──│ Router │ │ │
│ │ │ (BGP) │ │ (BGP) │ │ (BGP) │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ │ │ │ │ │ │
│ │ └────────────────┼────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │
│ │ │ L4 LOAD BALANCER │ │ │
│ │ │ │ │ │
│ │ │ • ECMP (Equal-Cost Multi-Path) hashing │ │ │
│ │ │ • Connection tracking for session affinity │ │ │
│ │ │ • DDoS mitigation (SYN flood protection) │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ SERVER LAYER │ │
│ │ │ │
│ │ ┌───────────────────────────────────────────────────────────────┐ │ │
│ │ │ HTTP PROXY │ │ │
│ │ │ │ │ │
│ │ │ • TLS termination (hardware accelerated) │ │ │
│ │ │ • HTTP/2 + HTTP/3 (QUIC) support │ │ │
│ │ │ • Request routing (host header → backend) │ │ │
│ │ │ • Compression (Brotli, gzip) │ │ │
│ │ │ • Header manipulation │ │ │
│ │ └───────────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌───────────────────────────────────────────────────────────────┐ │ │
│ │ │ CACHE LAYER │ │ │
│ │ │ │ │ │
│ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │
│ │ │ │ RAM │ │ SSD │ │ SSD │ │ HDD │ │ │ │
│ │ │ │ Cache │──│ Cache │──│ Cache │──│ Cache │ │ │ │
│ │ │ │ (Hot) │ │ (Warm) │ │ (Warm) │ │ (Cold) │ │ │ │
│ │ │ │ ~100GB │ │ ~2TB │ │ ~2TB │ │ ~20TB │ │ │ │
│ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │
│ │ │ │ │ │
│ │ │ Cache Hierarchy: │ │ │
│ │ │ • L1 (RAM): Sub-millisecond, frequently accessed │ │ │
│ │ │ • L2 (SSD): 1-5ms, warm content │ │ │
│ │ │ • L3 (HDD): 5-20ms, long-tail content │ │ │
│ │ └───────────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ │ Cache Miss │ │
│ │ ▼ │ │
│ │ ┌───────────────────────────────────────────────────────────────┐ │ │
│ │ │ ORIGIN FETCH │ │ │
│ │ │ │ │ │
│ │ │ Options: │ │ │
│ │ │ 1. Direct to origin (if no tiered cache) │ │ │
│ │ │ 2. Through origin shield (reduces origin load) │ │ │
│ │ │ 3. Through regional tier (hierarchical caching) │ │ │
│ │ └───────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Caching Architecture
Cache Key Design
The cache key determines what constitutes a "unique" cacheable object. Poor cache key design is the #1 cause of low hit ratios.
┌─────────────────────────────────────────────────────────────────────────────┐
│ CACHE KEY ANATOMY │
│ │
│ Request: GET https://cdn.example.com/images/hero.jpg?v=2&tracking=abc123 │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ DEFAULT CACHE KEY │ │
│ │ │ │
│ │ Components: │ │
│ │ • Scheme: https │ │
│ │ • Host: cdn.example.com │ │
│ │ • Path: /images/hero.jpg │ │
│ │ • Query: v=2&tracking=abc123 │ │
│ │ │ │
│ │ Full Key: https://cdn.example.com/images/hero.jpg?v=2&tracking=abc │ │
│ │ │ │
│ │ Problem: tracking parameter creates unique key per user! │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ OPTIMIZED CACHE KEY │ │
│ │ │ │
│ │ Configuration: │ │
│ │ • Include query params: v (version) │ │
│ │ • Exclude query params: tracking, utm_*, fbclid, etc. │ │
│ │ • Normalize: lowercase host, sort query params │ │
│ │ │ │
│ │ Optimized Key: https://cdn.example.com/images/hero.jpg?v=2 │ │
│ │ │ │
│ │ Result: Same asset → same cache key → higher hit ratio │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Cache Key Variations:
// CDN configuration for cache key normalization
interface CacheKeyConfig {
// Query string handling
queryString: {
include?: string[]; // Whitelist specific params
exclude?: string[]; // Blacklist specific params
includeAll?: boolean; // Include all (default true)
sort?: boolean; // Alphabetize params (recommended)
};
// Header-based variations
headers: {
include?: string[]; // Vary cache by these headers
// Common: Accept-Encoding, Accept-Language
// Avoid: Cookie, Authorization (kills cache)
};
// Cookie-based variations
cookies: {
include?: string[]; // Vary by specific cookies
// e.g., country_code, ab_variant
// Never include session cookies
};
// Device detection
device?: {
vary?: boolean; // Separate mobile/desktop caches
// Creates 2-3x cache storage but enables device-specific content
};
}
// Example: Cloudflare Cache Key configuration
const cacheRules = {
'/api/*': {
queryString: { includeAll: true, sort: true },
headers: { include: ['Accept'] },
cookies: { include: [] } // No cookie variation for API
},
'/static/*': {
queryString: { include: ['v', 'hash'] }, // Only version params
headers: { include: ['Accept-Encoding'] },
cookies: { include: [] }
},
'/personalized/*': {
queryString: { includeAll: true },
headers: { include: ['Accept-Encoding'] },
cookies: { include: ['user_segment'] } // Vary by segment, not user
}
};
Tiered Caching (Origin Shield)
Without tiered caching, every cache miss at every PoP hits origin. With 200 PoPs and a 95% hit ratio, that's still 200 * 0.05 = 10 origin requests per cache fill.
┌─────────────────────────────────────────────────────────────────────────────┐
│ WITHOUT ORIGIN SHIELD │
│ │
│ Cache Miss at Tokyo ──────────────────────────────▶ Origin │
│ Cache Miss at Singapore ──────────────────────────────▶ Origin │
│ Cache Miss at Mumbai ──────────────────────────────▶ Origin │
│ Cache Miss at Frankfurt ──────────────────────────────▶ Origin │
│ Cache Miss at London ──────────────────────────────▶ Origin │
│ │
│ Result: 5 requests to origin for the same content │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ WITH ORIGIN SHIELD │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ EDGE TIER │ │
│ │ │ │
│ │ Tokyo ──┐ │ │
│ │ │ │ │
│ │ Singapore ──┼──▶ APAC Shield (Singapore) ─┐ │ │
│ │ │ │ │ │
│ │ Mumbai ─────┘ │ │ │
│ │ │ │ │
│ │ Frankfurt ──┬──▶ EU Shield (Frankfurt) ───┼──▶ Origin │ │
│ │ │ │ │ │
│ │ London ─────┘ │ │ │
│ │ │ │ │
│ │ US-East ────┬──▶ US Shield (Ashburn) ─────┘ │ │
│ │ │ │ │
│ │ US-West ────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Result: 1-3 requests to origin (one per shield region) │
│ │
│ Shield Benefits: │
│ • Reduces origin load by 10-100x │
│ • Shields absorb traffic spikes │
│ • Better cache efficiency (fewer unique misses) │
│ │
│ Shield Tradeoffs: │
│ • Additional latency hop (50-100ms) on shield miss │
│ • Shield PoP becomes SPOF for region │
│ • Increased complexity in cache invalidation │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Cache Fill Optimization
sequenceDiagram
participant U as User
participant E as Edge PoP
participant S as Shield
participant O as Origin
U->>E: Request /asset.js
E->>E: Cache lookup (MISS)
Note over E: Request Coalescing<br/>Multiple requests for same<br/>asset collapse into one
E->>S: Forward to shield
S->>S: Cache lookup (MISS)
S->>O: Single origin request
O->>S: Response (200 OK)
Note over S: Cache at shield<br/>TTL: 1 hour
S->>E: Response with cache headers
Note over E: Cache at edge<br/>TTL: 1 hour
E->>U: Response
Note over E: Subsequent requests<br/>served from edge cache
Request Coalescing Implementation:
// Pseudo-code for request coalescing at CDN edge
class CacheNode {
private cache: Map<string, CachedResponse> = new Map();
private inflight: Map<string, Promise<Response>> = new Map();
async fetch(request: Request): Promise<Response> {
const cacheKey = this.computeCacheKey(request);
// Check cache
const cached = this.cache.get(cacheKey);
if (cached && !this.isStale(cached)) {
return cached.response.clone();
}
// Check if request already in flight
const pending = this.inflight.get(cacheKey);
if (pending) {
// Wait for existing request instead of making duplicate
const response = await pending;
return response.clone();
}
// First request - fetch and cache
const fetchPromise = this.fetchFromUpstream(request)
.then(response => {
// Cache if cacheable
if (this.isCacheable(response)) {
this.cache.set(cacheKey, {
response: response.clone(),
timestamp: Date.now(),
ttl: this.extractTTL(response)
});
}
return response;
})
.finally(() => {
// Clean up inflight after small delay
// Delay catches near-simultaneous requests
setTimeout(() => this.inflight.delete(cacheKey), 100);
});
this.inflight.set(cacheKey, fetchPromise);
return fetchPromise;
}
private isStale(cached: CachedResponse): boolean {
return Date.now() > cached.timestamp + cached.ttl;
}
}
Cache Control Headers
Header Semantics
┌─────────────────────────────────────────────────────────────────────────────┐
│ CACHE-CONTROL DIRECTIVES │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ STORAGE DIRECTIVES │ │
│ │ │ │
│ │ public │ Any cache can store (CDN, browser, proxies) │ │
│ │ private │ Only browser can cache (user-specific content) │ │
│ │ no-store │ Never cache anywhere (truly dynamic/sensitive) │ │
│ │ no-cache │ Store but revalidate before use (misleading name) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ EXPIRATION DIRECTIVES │ │
│ │ │ │
│ │ max-age=N │ Fresh for N seconds from response │ │
│ │ s-maxage=N │ Fresh for N seconds at shared caches only │ │
│ │ stale-while-revalidate=N │ Serve stale for N sec while fetching │ │
│ │ stale-if-error=N │ Serve stale for N sec if origin errors │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ REVALIDATION DIRECTIVES │ │
│ │ │ │
│ │ must-revalidate │ Must check origin when stale (no stale) │ │
│ │ proxy-revalidate │ must-revalidate for shared caches only │ │
│ │ immutable │ Never revalidate (content hash in URL) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ CDN-SPECIFIC HEADERS │ │
│ │ │ │
│ │ Surrogate-Control │ Instructions for CDN only (stripped) │ │
│ │ CDN-Cache-Control │ Cloudflare-specific CDN instructions │ │
│ │ Cache-Tag │ Tag-based invalidation (Cloudflare/Fastly) │ │
│ │ Surrogate-Key │ Tag-based invalidation (Fastly) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Cache Strategy Patterns
// Production cache header configurations
interface CacheStrategy {
pattern: string;
headers: Record<string, string>;
rationale: string;
}
const strategies: CacheStrategy[] = [
{
pattern: '/static/[hash]/*',
headers: {
'Cache-Control': 'public, max-age=31536000, immutable',
'CDN-Cache-Control': 'max-age=31536000'
},
rationale: 'Content-hashed static assets. URL changes when content changes. Cache forever.'
},
{
pattern: '/api/public/*',
headers: {
'Cache-Control': 'public, max-age=60, stale-while-revalidate=600, stale-if-error=86400',
'CDN-Cache-Control': 'max-age=60',
'Cache-Tag': 'api,public'
},
rationale: 'Public API data. Short TTL with SWR for freshness without latency penalty.'
},
{
pattern: '/api/user/*',
headers: {
'Cache-Control': 'private, no-store',
'CDN-Cache-Control': 'no-store'
},
rationale: 'User-specific data. Never cache at CDN, may cache in browser with care.'
},
{
pattern: '/*.html',
headers: {
'Cache-Control': 'public, max-age=0, must-revalidate',
'CDN-Cache-Control': 'max-age=300',
'Cache-Tag': 'html,pages'
},
rationale: 'HTML pages. Browsers must revalidate, CDN caches 5 min. Can instant purge via tags.'
},
{
pattern: '/images/*',
headers: {
'Cache-Control': 'public, max-age=86400, stale-while-revalidate=604800',
'CDN-Cache-Control': 'max-age=604800',
'Cache-Tag': 'images'
},
rationale: 'Images. Long CDN cache (1 week), browser caches 1 day, can serve stale 1 week.'
},
{
pattern: '/fonts/*',
headers: {
'Cache-Control': 'public, max-age=31536000, immutable',
'CDN-Cache-Control': 'max-age=31536000',
'Access-Control-Allow-Origin': '*'
},
rationale: 'Fonts. Immutable, cache forever. CORS for cross-origin font loading.'
}
];
Vary Header Complexity
The Vary header specifies which request headers affect the cache key. Misuse destroys cache efficiency.
┌─────────────────────────────────────────────────────────────────────────────┐
│ VARY HEADER │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ GOOD: Vary: Accept-Encoding │ │
│ │ │ │
│ │ Creates predictable variants: │ │
│ │ • /asset.js (Accept-Encoding: identity) │ │
│ │ • /asset.js (Accept-Encoding: gzip) │ │
│ │ • /asset.js (Accept-Encoding: br) │ │
│ │ │ │
│ │ 3 cached copies max. Efficient. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ BAD: Vary: Cookie │ │
│ │ │ │
│ │ Creates unbounded variants: │ │
│ │ • /page.html (Cookie: session=abc123...) │ │
│ │ • /page.html (Cookie: session=def456...) │ │
│ │ • /page.html (Cookie: session=ghi789...) │ │
│ │ • ... millions more │ │
│ │ │ │
│ │ Effectively uncacheable. 0% hit ratio. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ ACCEPTABLE: Vary: Accept-Language │ │
│ │ │ │
│ │ Creates bounded but numerous variants: │ │
│ │ • /page.html (Accept-Language: en-US) │ │
│ │ • /page.html (Accept-Language: es) │ │
│ │ • /page.html (Accept-Language: zh-CN) │ │
│ │ • ... ~50-100 languages │ │
│ │ │ │
│ │ Acceptable if you actually serve different content per language. │ │
│ │ Better: Use distinct URLs (/en/page, /es/page) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Cache Invalidation
The Fundamental Problem
Cache invalidation is hard because caches are distributed. There's no global lock, no atomic broadcast, no guaranteed ordering.
┌─────────────────────────────────────────────────────────────────────────────┐
│ CACHE INVALIDATION CHALLENGES │
│ │
│ Timeline of Distributed Invalidation: │
│ │
│ T=0ms: Origin content updated │
│ T=1ms: Invalidation request sent │
│ T=50ms: US-East edge receives invalidation │
│ T=80ms: EU-West edge receives invalidation │
│ T=150ms: APAC edge receives invalidation │
│ T=200ms: South America edge receives invalidation │
│ │
│ Problem: During T=1ms to T=200ms, different users see different content │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ User A (US-East, T=70ms): Sees NEW content (invalidation arrived) │ │
│ │ User B (APAC, T=70ms): Sees OLD content (invalidation pending) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ This is inherent to distributed systems. No CDN can provide instant │
│ global consistency. Design for eventual consistency. │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Invalidation Strategies
// 1. URL-based purge
// Simple but doesn't scale for many objects
await cdn.purge({ url: 'https://cdn.example.com/asset.js' });
// 2. Prefix purge
// Purge all URLs matching prefix
await cdn.purge({ prefix: 'https://cdn.example.com/api/' });
// 3. Tag-based purge (Surrogate-Key / Cache-Tag)
// Most flexible - purge by logical grouping
await cdn.purge({ tags: ['product-123', 'category-electronics'] });
// 4. Purge everything (nuclear option)
await cdn.purge({ all: true });
Tag-Based Invalidation Architecture:
┌─────────────────────────────────────────────────────────────────────────────┐
│ TAG-BASED INVALIDATION │
│ │
│ Origin Response: │
│ HTTP/1.1 200 OK │
│ Cache-Control: public, max-age=3600 │
│ Cache-Tag: product-123, category-electronics, page-home │
│ Content-Type: text/html │
│ │
│ CDN stores: │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ URL: /products/123 │ │
│ │ Tags: [product-123, category-electronics, page-home] │ │
│ │ Content: <html>...</html> │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Tag Index (reverse lookup): │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ product-123: [/products/123, /cart?items=123, ...] │ │
│ │ category-electronics: [/products/123, /category/electronics, ...] │ │
│ │ page-home: [/products/123, /, /featured, ...] │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Purge by tag: │
│ await cdn.purge({ tags: ['product-123'] }); │
│ │
│ Result: All URLs tagged with 'product-123' are invalidated │
│ • /products/123 │
│ • /cart?items=123 │
│ • Any other page that includes this product │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Invalidation Propagation
sequenceDiagram
participant A as Admin
participant API as CDN API
participant C as Control Plane
participant E1 as Edge (US-East)
participant E2 as Edge (EU)
participant E3 as Edge (APAC)
A->>API: Purge tag: "product-123"
API->>C: Validate & enqueue
par Fan-out to edges
C->>E1: Invalidation message
C->>E2: Invalidation message
C->>E3: Invalidation message
end
E1->>E1: Remove matching entries
E2->>E2: Remove matching entries
E3->>E3: Remove matching entries
Note over E1,E3: Propagation complete<br/>~100-500ms globally
E1-->>C: ACK
E2-->>C: ACK
E3-->>C: ACK
C-->>API: Purge complete
API-->>A: 200 OK
Soft Purge vs Hard Purge
┌─────────────────────────────────────────────────────────────────────────────┐
│ SOFT VS HARD PURGE │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ HARD PURGE │ │
│ │ │ │
│ │ Before: /asset.js in cache, valid for 1 hour │ │
│ │ Purge: Delete entry │ │
│ │ After: /asset.js not in cache │ │
│ │ │ │
│ │ Next request: Cache MISS → fetch from origin │ │
│ │ │ │
│ │ Risk: If origin is down, users get errors │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ SOFT PURGE │ │
│ │ │ │
│ │ Before: /asset.js in cache, valid for 1 hour │ │
│ │ Purge: Mark as stale (but keep) │ │
│ │ After: /asset.js in cache, marked stale │ │
│ │ │ │
│ │ Next request: Serve stale + background revalidate │ │
│ │ │ │
│ │ Benefit: Users always get a response, even if origin slow/down │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Recommendation: Use soft purge for non-critical updates │
│ Use hard purge for security/compliance │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Performance Optimization
Connection Optimization
┌─────────────────────────────────────────────────────────────────────────────┐
│ CDN CONNECTION OPTIMIZATIONS │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TLS OPTIMIZATION │ │
│ │ │ │
│ │ Without Optimization: │ │
│ │ ├── DNS lookup: 50ms │ │
│ │ ├── TCP handshake: 1 RTT (100ms) │ │
│ │ ├── TLS handshake: 2 RTT (200ms) │ │
│ │ ├── HTTP request/response: 1 RTT (100ms) │ │
│ │ └── Total: 450ms minimum │ │
│ │ │ │
│ │ With CDN Optimization: │ │
│ │ ├── DNS: Cached in browser (0ms) │ │
│ │ ├── TCP: Already connected via Keep-Alive (0ms) │ │
│ │ ├── TLS: Session resumption (0ms) or 0-RTT (QUIC) │ │
│ │ ├── HTTP/2: Multiplexed on existing connection (0ms setup) │ │
│ │ ├── HTTP request/response: 1 RTT (20ms to edge) │ │
│ │ └── Total: 20ms │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ HTTP/2 MULTIPLEXING │ │
│ │ │ │
│ │ HTTP/1.1 (6 connections max per host): │ │
│ │ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐ │ │
│ │ │req1│ │req2│ │req3│ │req4│ │req5│ │req6│ req7 waiting... │ │
│ │ └────┘ └────┘ └────┘ └────┘ └────┘ └────┘ │ │
│ │ │ │
│ │ HTTP/2 (single connection, unlimited streams): │ │
│ │ ┌────────────────────────────────────────────────────────────────┐│ │
│ │ │ stream1 │ stream2 │ stream3 │ stream4 │ stream5 │ stream6 │ ...││ │
│ │ └────────────────────────────────────────────────────────────────┘│ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ HTTP/3 (QUIC) │ │
│ │ │ │
│ │ Advantages over HTTP/2: │ │
│ │ • 0-RTT connection resumption │ │
│ │ • No head-of-line blocking (per-stream flow control) │ │
│ │ • Connection migration (change IP without reconnect) │ │
│ │ • Built-in encryption (TLS 1.3 integrated) │ │
│ │ │ │
│ │ Particularly beneficial on mobile (network switching) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Compression Strategy
// Compression configuration at CDN
interface CompressionConfig {
// Brotli: Better compression, slower encode
brotli: {
enabled: true,
quality: 5, // 1-11, higher = smaller but slower
minSize: 1024, // Don't compress small files
types: [
'text/html',
'text/css',
'text/javascript',
'application/javascript',
'application/json',
'image/svg+xml'
]
},
// Gzip: Fallback for older clients
gzip: {
enabled: true,
level: 6, // 1-9
minSize: 1024,
types: [...] // Same as brotli
}
}
// Compression is applied at edge, not origin
// Origin sends uncompressed, CDN compresses once and caches compressed versions
// Response headers:
// Vary: Accept-Encoding
// Content-Encoding: br (or gzip)
Compression Tradeoffs:
| Format | Compression Ratio | Encode Speed | Decode Speed | Browser Support |
|---|---|---|---|---|
| None | 1x | N/A | N/A | 100% |
| Gzip | 3-4x | Fast | Fast | 99%+ |
| Brotli | 4-5x | Slow | Fast | 95%+ |
| Zstd | 4-5x | Fast | Fast | Growing |
Image Optimization
┌─────────────────────────────────────────────────────────────────────────────┐
│ CDN IMAGE OPTIMIZATION │
│ │
│ Request: /images/hero.jpg?w=800&format=auto&quality=80 │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TRANSFORMATION PIPELINE │ │
│ │ │ │
│ │ 1. Origin Fetch (if not cached) │ │
│ │ └── Fetch original: hero.jpg (5MB, 4000x3000) │ │
│ │ │ │
│ │ 2. Resize │ │
│ │ └── Scale to width=800, maintain aspect ratio │ │
│ │ └── Result: 800x600 │ │
│ │ │ │
│ │ 3. Format Selection (format=auto) │ │
│ │ └── Check Accept header │ │
│ │ └── AVIF > WebP > JPEG (based on browser support) │ │
│ │ │ │
│ │ 4. Quality Adjustment │ │
│ │ └── Apply quality=80 compression │ │
│ │ │ │
│ │ 5. Cache transformed result │ │
│ │ └── Cache key: /images/hero.jpg?w=800&format=webp&quality=80 │ │
│ │ └── Size: 45KB (vs 5MB original) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Format Comparison (same visual quality): │
│ ┌───────────────────────────────────────────────────────────────────┐ │
│ │ Format │ Size │ Browser Support │ Use Case │ │
│ │──────────│─────────│──────────────────│──────────────────────────│ │
│ │ JPEG │ 100KB │ 100% │ Fallback │ │
│ │ WebP │ 65KB │ 96% │ Default modern │ │
│ │ AVIF │ 45KB │ 85% │ Best quality/size │ │
│ └───────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Security at the CDN Layer
DDoS Mitigation
┌─────────────────────────────────────────────────────────────────────────────┐
│ CDN DDoS PROTECTION │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 3/4 PROTECTION │ │
│ │ │ │
│ │ Attack: Volumetric (10 Tbps SYN flood) │ │
│ │ │ │
│ │ Defense: │ │
│ │ 1. Anycast absorption (distribute across 300 PoPs) │ │
│ │ 2. BGP Flowspec (filter at network edge) │ │
│ │ 3. SYN cookies (stateless SYN handling) │ │
│ │ 4. Rate limiting per source IP │ │
│ │ │ │
│ │ Result: 10 Tbps becomes 33 Gbps per PoP (absorbable) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 7 PROTECTION │ │
│ │ │ │
│ │ Attack: HTTP flood (1M requests/sec to dynamic endpoint) │ │
│ │ │ │
│ │ Defense: │ │
│ │ 1. Rate limiting (per IP, per session, per endpoint) │ │
│ │ 2. Bot detection (JavaScript challenge, CAPTCHA) │ │
│ │ 3. WAF rules (block known attack patterns) │ │
│ │ 4. Reputation scoring (block known-bad IPs/ASNs) │ │
│ │ 5. Serve from cache (origin never sees traffic) │ │
│ │ │ │
│ │ Result: 1M rps absorbed, origin sees <1000 rps │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
WAF at Edge
// WAF rule configuration
const wafRules = {
// OWASP Core Rule Set patterns
sqlInjection: {
enabled: true,
action: 'block',
patterns: [
/(\%27)|(\')|(\-\-)|(\%23)|(#)/i,
/\b(union|select|insert|update|delete|drop|truncate)\b/i
]
},
xss: {
enabled: true,
action: 'block',
patterns: [
/<script\b[^<]*(?:(?!<\/script>)<[^<]*)*<\/script>/gi,
/javascript:/i,
/on\w+\s*=/i
]
},
rateLimit: {
enabled: true,
rules: [
{ path: '/api/login', limit: 5, window: 60, action: 'challenge' },
{ path: '/api/*', limit: 1000, window: 60, action: 'block' },
{ path: '/*', limit: 10000, window: 60, action: 'block' }
]
},
geoBlocking: {
enabled: true,
blockedCountries: ['XX', 'YY'], // Embargoed countries
action: 'block'
},
botManagement: {
enabled: true,
challengeThreshold: 30, // Bot score 0-100
blockThreshold: 10
}
};
Origin Protection
┌─────────────────────────────────────────────────────────────────────────────┐
│ ORIGIN PROTECTION │
│ │
│ Problem: Attackers bypass CDN by hitting origin directly │
│ │
│ Solutions: │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 1. IP ALLOWLISTING │ │
│ │ │ │
│ │ Origin firewall only accepts traffic from CDN IP ranges: │ │
│ │ • Cloudflare: 173.245.48.0/20, 103.21.244.0/22, ... │ │
│ │ • Fastly: 23.235.32.0/20, 43.249.72.0/22, ... │ │
│ │ │ │
│ │ Risk: CDN IP ranges are public. Attacker can spoof. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 2. AUTHENTICATED ORIGIN PULLS │ │
│ │ │ │
│ │ CDN includes secret header on origin requests: │ │
│ │ X-Origin-Auth: sha256(timestamp + secret) │ │
│ │ │ │
│ │ Origin validates header before responding. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ 3. PRIVATE NETWORK / TUNNEL │ │
│ │ │ │
│ │ Origin not on public internet at all. │ │
│ │ CDN connects via private tunnel (Cloudflare Tunnel, Argo, etc.) │ │
│ │ │ │
│ │ Best security: Origin has no public IP to attack. │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Observability
CDN Metrics
// Key CDN metrics to monitor
interface CDNMetrics {
// Traffic metrics
requestsPerSecond: number;
bandwidthBytesPerSecond: number;
uniqueVisitors: number;
// Cache performance
cacheHitRatio: number; // Target: >95%
cacheHitRatioByStatus: {
hit: number; // Served from edge cache
miss: number; // Fetched from origin
stale: number; // Served stale while revalidating
revalidated: number; // Revalidated with origin
expired: number; // Expired, fetched fresh
};
// Latency
edgeLatencyP50: number; // Target: <20ms
edgeLatencyP99: number; // Target: <100ms
originLatencyP50: number;
originLatencyP99: number;
ttfbP50: number;
ttfbP99: number;
// Errors
errorRate: number; // Target: <0.1%
errorsByType: {
'4xx': number;
'5xx': number;
originTimeout: number;
originUnreachable: number;
};
// Security
blockedRequests: number;
wafTriggered: number;
ddosEventsActive: number;
}
Real-Time Logging
┌─────────────────────────────────────────────────────────────────────────────┐
│ CDN LOG PIPELINE │
│ │
│ Edge Request ──▶ Log Entry Generated │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ LOG FORMAT │ │
│ │ │ │
│ │ { │ │
│ │ "timestamp": "2024-01-15T10:30:00.000Z", │ │
│ │ "clientIP": "203.0.113.50", │ │
│ │ "clientCountry": "JP", │ │
│ │ "edgeLocation": "NRT", │ │
│ │ "request": { │ │
│ │ "method": "GET", │ │
│ │ "host": "cdn.example.com", │ │
│ │ "path": "/images/hero.jpg", │ │
│ │ "protocol": "HTTP/2", │ │
│ │ "userAgent": "Mozilla/5.0..." │ │
│ │ }, │ │
│ │ "response": { │ │
│ │ "status": 200, │ │
│ │ "bytes": 45000, │ │
│ │ "cacheStatus": "HIT", │ │
│ │ "contentType": "image/webp" │ │
│ │ }, │ │
│ │ "timing": { │ │
│ │ "edgeMs": 5, │ │
│ │ "originMs": null, │ │
│ │ "ttfbMs": 8 │ │
│ │ }, │ │
│ │ "security": { │ │
│ │ "wafAction": null, │ │
│ │ "botScore": 95, │ │
│ │ "threatScore": 0 │ │
│ │ } │ │
│ │ } │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ LOG DESTINATIONS │ │
│ │ │ │
│ │ • Real-time streaming to Kafka/Kinesis │ │
│ │ • Batch upload to S3/R2/GCS │ │
│ │ • Direct to SIEM (Splunk, Elastic) │ │
│ │ • Analytics warehouse (BigQuery, Snowflake) │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Production Incidents
Incident 1: Cache Stampede on TTL Expiry
Scenario: High-traffic asset with 60-second TTL. At expiry, 1000 concurrent requests hit origin simultaneously.
Impact: Origin overwhelmed, cascading failures, 5-minute outage.
Root Cause Analysis:
Timeline:
T=0:00 Cache entry expires at all edge PoPs simultaneously
T=0:01 1000 cache misses per PoP = 300,000 requests to origin
T=0:02 Origin connection pool exhausted
T=0:03 Origin starts 503ing
T=0:04 503s cached briefly (negative caching)
T=0:05 Cycle repeats as negative cache expires
Fix:
// 1. Request coalescing (CDN-side)
// Most CDNs do this automatically, but verify configuration
// 2. Stale-while-revalidate
// Origin response:
const headers = {
'Cache-Control': 'public, max-age=60, stale-while-revalidate=600'
};
// Result: Serves stale for up to 10 minutes while one request revalidates
// 3. Cache TTL jitter
// Add random offset to prevent synchronized expiry
function getCacheTTL(baseTTL: number): number {
const jitter = baseTTL * 0.1 * (Math.random() - 0.5); // ±5%
return Math.floor(baseTTL + jitter);
}
// 4. Origin-side rate limiting
// Even with CDN, protect origin from bursts
app.use('/api', rateLimit({
windowMs: 1000,
max: 100, // 100 requests per second max
message: { error: 'Rate limited', retryAfter: 1 }
}));
Incident 2: Cache Poisoning via Host Header
Scenario: Attacker sent requests with forged Host header. CDN cached response keyed by attacker's Host, but content included Host in URLs.
Impact: Legitimate users received pages with links to attacker's domain.
Attack:
GET /page HTTP/1.1
Host: evil.com <-- Forged
X-Forwarded-Host: cdn.example.com
Response cached with key including Host: evil.com
Content includes: <a href="https://evil.com/login">Login</a>
Fix:
// 1. Validate Host header at edge
export default {
async fetch(request: Request): Promise<Response> {
const host = request.headers.get('Host');
const allowedHosts = ['cdn.example.com', 'www.example.com'];
if (!host || !allowedHosts.includes(host)) {
return new Response('Invalid Host', { status: 400 });
}
// Continue processing
}
};
// 2. Don't include Host in response body
// Use relative URLs or configured base URL
const baseUrl = process.env.CANONICAL_BASE_URL; // Not from request
// 3. Configure CDN to normalize Host header
// Most CDNs can strip or override Host before caching
Incident 3: Negative Caching Propagation
Scenario: Brief origin outage (30 seconds). CDN cached 503 errors with default TTL. Users saw errors for minutes after origin recovered.
Fix:
// 1. Don't cache error responses (or use very short TTL)
// CDN configuration:
const cacheRules = {
errorPages: {
'5xx': { ttl: 0 }, // Don't cache
'404': { ttl: 60 } // Brief cache OK for 404
}
};
// 2. Use stale-if-error
const headers = {
'Cache-Control': 'public, max-age=60, stale-if-error=3600'
};
// Serves stale content if origin returns error
// 3. Health-based routing
// CDN checks origin health and routes to backup if primary fails
Tradeoffs and Design Decisions
CDN Provider Selection
| Factor | Cloudflare | AWS CloudFront | Fastly | Akamai |
|---|---|---|---|---|
| PoP Count | 300+ | 400+ | 90+ | 4000+ |
| Pricing | Flat bandwidth | Per-GB egress | Per-request | Enterprise |
| Edge Compute | Workers (powerful) | Lambda@Edge (limited) | Compute@Edge | EdgeWorkers |
| Purge Speed | <5s | ~15-60s | <150ms | ~5s |
| DDoS Included | Yes | Shield ($3K/mo) | Limited | Yes (enterprise) |
| Best For | Full-stack edge | AWS integration | Real-time purge | Enterprise/media |
Multi-CDN Strategy
┌─────────────────────────────────────────────────────────────────────────────┐
│ MULTI-CDN ARCHITECTURE │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ TRAFFIC MANAGER │ │
│ │ │ │
│ │ DNS-based or Anycast-based routing to multiple CDNs │ │
│ │ │ │
│ │ Routing Strategies: │ │
│ │ • Performance: Route to fastest CDN per region │ │
│ │ • Cost: Route to cheapest CDN meeting SLA │ │
│ │ • Reliability: Active-active across CDNs │ │
│ │ • Failover: Primary CDN with automatic failover │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────┼───────────┐ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ CDN A │ │ CDN B │ │ CDN C │ │
│ │ (60%) │ │ (30%) │ │ (10%) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ Benefits: │
│ • No single vendor lock-in │
│ • Leverage each CDN's strengths │
│ • Automatic failover if one CDN has issues │
│ • Negotiate better pricing with competition │
│ │
│ Costs: │
│ • Cache efficiency drops (same content cached multiple times) │
│ • Operational complexity (manage multiple configs) │
│ • Invalidation must hit all CDNs │
│ • Inconsistent features across CDNs │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Edge Compute vs Origin Compute
Decision Matrix:
┌────────────────────────────────────────────────────────────────────────────┐
│ Use Case │ Edge │ Origin │ Rationale │
├──────────────────────────────┼──────┼────────┼─────────────────────────────┤
│ Static file serving │ ✓ │ │ Latency, bandwidth costs │
│ A/B test routing │ ✓ │ │ No origin needed │
│ Geo-redirect │ ✓ │ │ Decision based on location │
│ JWT validation │ ✓ │ │ Fail fast, protect origin │
│ Image resizing │ ✓ │ │ Compute once, cache result │
│ Bot detection │ ✓ │ │ Block before origin │
│ Rate limiting │ ✓ │ │ Protect origin │
│ Header manipulation │ ✓ │ │ Simple transform │
├──────────────────────────────┼──────┼────────┼─────────────────────────────┤
│ Database queries │ │ ✓ │ Data locality │
│ Complex business logic │ │ ✓ │ CPU/memory limits │
│ Transaction processing │ │ ✓ │ Consistency requirements │
│ Large data processing │ │ ✓ │ Memory limits │
│ Long-running operations │ │ ✓ │ Execution time limits │
└────────────────────────────────────────────────────────────────────────────┘
Summary
A CDN is not a caching proxy—it's a globally distributed system with inherent complexity. Understanding this complexity is essential for designing systems that leverage CDN strengths while avoiding common pitfalls.
Key Architectural Insights:
-
Cache keys determine everything - Poor cache key design (e.g., including tracking parameters) destroys hit ratios. Audit your cache keys.
-
Origin shield is essential at scale - Without tiered caching, every PoP's cache miss hits origin. Shield reduces origin load 10-100x.
-
Invalidation is eventually consistent - There's no instant global purge. Design for brief inconsistency windows or use versioned URLs.
-
stale-while-revalidate is your friend - Eliminates cache stampedes and provides resilience during origin issues.
-
Vary header is dangerous -
Vary: Cookiedestroys cacheability. Minimize Vary usage. -
Security happens at the edge - DDoS mitigation, WAF, bot detection all benefit from edge enforcement.
-
Monitor cache hit ratio obsessively - It's the single most important CDN metric. Below 90%? Investigate.
-
Multi-CDN adds complexity - Only worth it for very large scale or strict availability requirements.
The difference between a naive CDN deployment and a well-architected one can be 10x in performance and cost. Invest the time to understand your CDN's behavior at the cache key level.
What did you think?