Wizard/docs/09-limits-and-performance.md

# 9. Limits, load control and performance

This chapter explains how Wizard keeps its resource use bounded, how it
protects itself against abuse, and what happens when it runs out of compute.
It covers all components, because the protection works in layers from Docker
down to the browser.

## Principles

1. **Running games always continue.** A game in progress is never stopped,
   slowed down on purpose or refused a move because of load.
2. **New work is refused when there is no capacity.** Creating lobbies and
   starting games is what adds load, so that is what gets refused — with a
   clear "server busy" message, not a timeout. (A waiting queue was considered
   and rejected as not worth the complexity.)
3. **Abuse is rejected immediately.** Floods and connection storms are cut off
   at the door, never queued.
4. **The host is protected no matter what.** Hard Docker caps guarantee that
   Wizard can never take CPU or memory from the other services on the machine,
   even if every software limit fails.
5. **Degrade, don't fail.** An overloaded or broken AI makes weaker moves (the
   heuristic, or a random move) instead of stopping games.

## Overview

```mermaid
flowchart TB
    subgraph docker["Layer 1: Docker caps (compose.yaml)"]
        direction TB
        subgraph server["wizard_server — 1 CPU / 512 MB"]
            U["uvicorn: max message 64 KB,<br/>max 1000 connections"]
            A["Layer 2: abuse limits<br/>connections, message rate,<br/>lobby creation (limits.py)"]
            L["Layer 3: load monitor +<br/>admission control (load.py)"]
            B["AI bridge: in-flight cap,<br/>circuit breaker (ai_bridge.py)"]
        end
        subgraph ai["wizard_ai — 1 CPU / 1 GB"]
            Q["Layer 4: workers, queue,<br/>deadlines, degradation<br/>(service/client.py)"]
        end
        W["wizard_website — 0.25 CPU / 64 MB"]
    end
    Browser -->|websocket| U --> A --> L
    L -->|AI turns| B -->|decide| Q
    L -. "server_status" .-> Browser
```

## Layer 1 — Docker caps

Set in `compose.yaml` from `.env` (defaults shown):

| Container | CPU | Memory | Processes |
|-----------|-----|--------|-----------|
| `wizard_server` | `SERVER_CPUS=1.0` | `SERVER_MEMORY=512M` | 256 |
| `wizard_ai` | `AI_CPUS=1.0` | `AI_MEMORY=1G` | 256 |
| `wizard_website` | `WEBSITE_CPUS=0.25` | `WEBSITE_MEMORY=64M` | 64 |

- **CPU** is a hard quota: when the container uses it up, the kernel
  throttles it. The server notices this (see layer 3) and refuses new games.
- **Memory** is a hard limit: a container exceeding it is killed by the kernel
  and restarted by Docker (`restart: unless-stopped`). For the server this
  would end all games (state is in memory), so the limit has lots of headroom:
  the server uses ~80 MB idle and ~100 MB with 100 games (see
  [measurements](#measurements)).
- **Processes** (`pids_limit`) stop fork bombs / thread explosions.
- The AI container sets `OMP_NUM_THREADS` / `MKL_NUM_THREADS` to `AI_WORKERS`,
  so numeric libraries (numpy, torch) don't start one thread per *host* core
  inside a 1-CPU container.

uvicorn (`wizard_server/Dockerfile`) adds transport limits:
`--ws-max-size 65536` (larger messages close the connection with code 1009;
real messages are < 1 KB), `--ws-max-queue 16`, `--limit-concurrency 1000`
(HTTP 503 beyond that — a last resort above the app's own connection limit).

## Layer 2 — abuse limits (server)

`wizard_server/app/limits.py`, applied in `WizardService.handle_client`
(`service.py`):

| Limit | Setting (default) | When exceeded |
|-------|-------------------|---------------|
| Open connections in total | `MAX_CONNECTIONS` (500) | `error` code `busy`, close **1013** "Try Again Later" |
| Open connections per client IP | `MAX_CONNECTIONS_PER_IP` (10) | same |
| Messages per connection | `MESSAGE_RATE` (10/s), `MESSAGE_BURST` (30) | message dropped; `error` code `rate_limited` (only on the 1st and every 10th drop) |
| Flooding | `MAX_RATE_VIOLATIONS` (50 drops) | connection closed with **1008** |
| Lobbies created per client IP | `LOBBY_CREATE_RATE` (10 per minute) | `error` code `rate_limited` |

Building blocks:

- `TokenBucket(rate, burst)` — the classic algorithm: a bucket holds up to
  `burst` tokens, refills at `rate` per second, each action takes one token.
  Allows short bursts (a player clicking quickly) but bounds the average.
- `KeyedRateLimiter` — one bucket per key (client IP); buckets that refilled
  completely are forgotten by the cleanup task every 30 s.
- `ConnectionLimiter` — counts open connections in total and per IP; released
  in the `finally` block of `handle_client`, so crashes cannot leak slots.

**Client IP**: `websocket.client.host`. Behind Caddy, uvicorn's
`--proxy-headers` sets it from `X-Forwarded-For`; Caddy ignores
client-supplied `X-Forwarded-For` values, so the IP cannot be spoofed. In
local development everything comes from `127.0.0.1`.

Normal play needs about one message per second, so these limits are never
noticeable for humans. Several players behind one NAT (e.g. a household)
share the per-IP connection limit — raise `MAX_CONNECTIONS_PER_IP` if that is
a realistic use case.

## Layer 3 — load monitor and admission control (server)

### Detecting overload (`load.py`)

`LoadMonitor.run()` is started in the app lifespan and takes a sample every
`LOAD_SAMPLE_INTERVAL` (1 s):

| Signal | How it is measured | Why |
|--------|--------------------|-----|
| **Event loop lag** | the monitor sleeps 1 s and measures how much later it woke up | the most direct measure of "the server reacts slowly"; works everywhere |
| **CPU** | process CPU time per second, relative to the container's CPU quota (at most 1 core, because game logic runs on one core) | detects saturation before players notice lag |
| **Throttling** | `nr_throttled / nr_periods` from `cpu.stat` | share of scheduling periods in which the container hit its Docker CPU cap |

CPU quota and throttling come from the process's **own cgroup** (cgroup v2):
`/proc/self/cgroup` gives its path, e.g. `0::/` inside a container (→
`/sys/fs/cgroup`) or `0::/user.slice/…scope` on a host. Without cgroup files
(e.g. macOS) the monitor still works with lag and CPU against one core.

Each sample is classified (`classify`), and `next_level` turns the recent
classifications into the current level with hysteresis:

| Level | Entered | Left |
|-------|---------|------|
| `normal` | — | — |
| `busy` | one sample with lag ≥ `LOAD_LAG_BUSY` (0.1 s) or CPU ≥ `LOAD_CPU_BUSY` (70 %) | after `LOAD_RECOVER_SAMPLES` (10) normal samples in a row |
| `overloaded` | `LOAD_OVERLOAD_SAMPLES` (3) samples in a row with lag ≥ `LOAD_LAG_OVERLOAD` (0.25 s), CPU ≥ `LOAD_CPU_OVERLOAD` (90 %) or throttled ≥ `LOAD_THROTTLE_OVERLOAD` (50 %) — **or one** sample with lag ≥ `LOAD_LAG_SEVERE` (1 s) | after 10 samples in a row that are not overloaded |

Why hysteresis: without it a server at the edge would flip between states
every second, and every flip is broadcast to all players. Why the "severe"
rule: when the event loop is extremely late, samples themselves arrive late
(one per several seconds), so waiting for three would delay detection a lot.

`busy` is informational only. `LOAD_FORCE_LEVEL` pins the level for testing.

### Admission control (`service.py`)

`WizardService.accepting_games` is true unless the level is `overloaded` or
`MAX_ACTIVE_GAMES` (50) games are running (`LobbyManager.active_games()`).

| Action | While not accepting games |
|--------|---------------------------|
| `create_lobby` | refused — `error` code `busy` |
| `start_game` | refused — `error` code `busy` |
| `join_lobby` | allowed (cheap) |
| moves in running games, `return_to_lobby`, leaving | allowed (protects running games) |
| reconnects (`hello` with token) | allowed |

Refusing new games sheds load by itself: finished games are not replaced, so
the load falls and the server recovers without intervention.

### Telling the players

Every connected client gets a `server_status` message right after `welcome`
and whenever the level or `accepting_games` changes:

```json
{"type": "server_status", "load": "overloaded", "accepting_games": false}
```

Changes come from the load monitor (`on_change`) and from lobbies
(`on_status_change` when a game starts, finishes or is abandoned), and are
de-duplicated so an unchanged status is not re-sent.

### Health endpoint

`GET /health` (public: `https://game.marvinkrausser.com/wizard/api/health`):

```json
{
  "status": "ok",
  "load": "normal", "loop_lag": 0.0003, "cpu": 0.012, "throttled": 0.0,
  "accepting_games": true,
  "lobbies": 4, "active_games": 3, "sessions": 9, "connections": 7,
  "ai_available": true,
  "ai": {"connections": 1, "pending": 0, "paused": false}
}
```

`status` **stays "ok" under load** on purpose: the Docker health check must
never mark an overloaded server unhealthy, because a restart would end every
running game.

## Layer 4 — AI cost bounds

### In the server (`ai_bridge.py`)

| Mechanism | Setting (default) | Behaviour |
|-----------|-------------------|-----------|
| Deadline | `AI_DECISION_TIMEOUT` (5 s) | sent as `timeout` in every request; no answer in time → random move |
| In-flight cap | `AI_MAX_INFLIGHT` (64) | if that many requests already wait on an AI connection, the move is played randomly without asking. Keep it ≥ `MAX_ACTIVE_GAMES`: every game has at most one AI deciding at a time |
| Circuit breaker | `AI_BREAKER_FAILURES` (5), `AI_BREAKER_COOLDOWN` (30 s) | after 5 **real failures** in a row (timeouts, lost connection, malformed/illegal answers, unexpected errors) the AI is not asked for 30 s; games play random AI moves instead of waiting 5 s per move |

What counts as a failure matters: the breaker is for a *broken or hanging* AI
service. Deliberate load shedding — the in-flight cap, or the AI answering
`overloaded` / `expired` — is **not** a failure: the service is healthy and
answered immediately, and pausing it would only make play worse.

### In the AI service (`wizard_ai/wizard_ai/service/client.py`)

```mermaid
flowchart TD
    R[decide request] --> Q{"in progress ≥<br/>AI_MAX_QUEUE?"}
    Q -- yes --> E1["error: overloaded"]
    Q -- no --> W[wait for a free worker<br/>AI_WORKERS]
    W --> X{"server timeout<br/>already passed?"}
    X -- yes --> E2["error: expired"]
    X -- no --> D{"≥ AI_DEGRADE_QUEUE others waiting,<br/>or primary agent slower than<br/>half the remaining time?"}
    D -- yes --> H[heuristic agent]
    D -- no --> M[primary agent / model]
    H --> OK["action + agent name"]
    M --> OK
```

| Setting (default) | Meaning |
|-------------------|---------|
| `AI_WORKERS` (1) | decisions computed at the same time; agents must be thread safe if > 1 |
| `AI_MAX_QUEUE` (64) | requests in progress (waiting + computing) before refusing with `overloaded` |
| `AI_DEGRADE_QUEUE` (8) | from this many other waiting requests, the heuristic decides instead of the model |

The service also keeps a smoothed average of the primary agent's decision
time; if it exceeds half of the time left before the server's deadline, the
heuristic answers instead. The reply's `agent` field says who decided (the
server logs it at debug level). With the heuristic as primary agent (today's
default without a trained model) there is nothing cheaper to fall back to, so
degradation only matters once a real model is deployed.

## What players see

| Situation | Start page | Lobby | Running game |
|-----------|-----------|-------|--------------|
| Normal | — | — | — |
| Not accepting games (overloaded or game limit) | notice "The server is busy…", *Create lobby* disabled, *Join* works | notice, *Start game* shows "Server busy" (disabled) | unchanged |
| Connection limit reached | red banner "The server is full right now. Retrying in a moment…", retries after 15 s, 30 s, 60 s … | — | — |
| Message rate exceeded | toast "Too many messages, slow down." | same | same |
| AI overloaded / broken | — | — | AI players make weaker (heuristic or random) moves |

Implemented in `useWizard.ts` (`serverStatus`, connection status `busy`),
`App.tsx`, `HomeScreen.tsx` and `LobbyScreen.tsx`.

## Configuration reference

All settings with their defaults. Server settings are read in
`wizard_server/app/config.py`, AI settings in `wizard_ai/wizard_ai/config.py`.
Settings marked *compose* are already wired through `compose.yaml` / `.env`;
the others can be added to the `environment:` section of the service.

| Variable | Component | Default | Wired in compose |
|----------|-----------|---------|------------------|
| `SERVER_CPUS` / `SERVER_MEMORY` | Docker | 1.0 / 512M | yes |
| `AI_CPUS` / `AI_MEMORY` | Docker | 1.0 / 1G | yes |
| `WEBSITE_CPUS` / `WEBSITE_MEMORY` | Docker | 0.25 / 64M | yes |
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | 500 / 10 | yes |
| `MESSAGE_RATE` / `MESSAGE_BURST` | server | 10 / 30 | no |
| `MAX_RATE_VIOLATIONS` | server | 50 | no |
| `LOBBY_CREATE_RATE` | server | 10 per minute | no |
| `MAX_LOBBIES` | server | 1000 | no |
| `MAX_ACTIVE_GAMES` | server | 50 | yes |
| `LOAD_SAMPLE_INTERVAL` | server | 1.0 s | no |
| `LOAD_LAG_BUSY` / `LOAD_LAG_OVERLOAD` / `LOAD_LAG_SEVERE` | server | 0.1 / 0.25 / 1.0 s | no |
| `LOAD_CPU_BUSY` / `LOAD_CPU_OVERLOAD` | server | 0.7 / 0.9 | no |
| `LOAD_THROTTLE_OVERLOAD` | server | 0.5 | no |
| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | server | 3 / 10 | no |
| `LOAD_FORCE_LEVEL` | server | empty | no |
| `AI_DECISION_TIMEOUT` | server | 5 s | no |
| `AI_MAX_INFLIGHT` | server | 64 | no |
| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | server | 5 / 30 s | no |
| `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | 1 / 64 / 8 | yes |

## Measurements

Measured on a development machine (uvicorn, heuristic AI, normal pacing:
`AI_MOVE_DELAY` 0.8 s, one scripted human per game taking 1.5 s per move,
4 players per game):

| Running games | Server CPU | Server memory | AI service CPU | AI memory |
|---------------|-----------|---------------|----------------|-----------|
| 0 | ~0 % | ~55–80 MB | ~0 % | ~30 MB |
| 50 | 0.8 % of a core | ~88 MB | 0.5 % | ~30 MB |
| 100 | 1.5 % of a core | ~99 MB | 0.9 % | ~30 MB |

Overload test: server limited to **15 % of one CPU**
(`systemd-run --user --scope -p CPUQuota=15%`), 30 games playing at full speed
(no delays, instant human moves):

| Time | Event |
|------|-------|
| 0 s | load starts |
| ~1 s | `busy` |
| ~3 s | `overloaded` (CPU at 100 % of quota, throttled 100 %) → `create_lobby` refused with `busy` |
| … | running games keep playing (534 moves made while overloaded) |
| ~14 s | games that finished could not restart → load gone → back to `normal` |

What this means:

- With realistic pacing a game costs almost nothing; **1 CPU is far more than
  50 games need**. `MAX_ACTIVE_GAMES=50` is a conservative safety cap, not a
  CPU requirement — raising it to a few hundred is fine with the heuristic AI.
- Overload only happens with abnormal traffic (scripts, floods) or a heavy AI
  model. That is exactly what the limits are for.
- A neural-network model will be the most expensive part. Measure its
  decision time and CPU with `python -m wizard_ai evaluate` and `docker stats`
  before raising the caps.

## Tuning guide

| Symptom | Adjust |
|---------|--------|
| "Server busy" although `/health` shows low `cpu` and `load: normal` | the game limit is reached: raise `MAX_ACTIVE_GAMES` (and `AI_MAX_INFLIGHT` to at least the same value) |
| "Server busy" with high `cpu` / `throttled` | give the server more CPU (`SERVER_CPUS`) — or accept it: the server is protecting itself |
| Friends behind one router get "server full" | raise `MAX_CONNECTIONS_PER_IP` |
| Busy/overloaded flips too eagerly on a slow machine | raise `LOAD_LAG_*` thresholds or `LOAD_OVERLOAD_SAMPLES` |
| AI plays weaker under load (logs: "Busy … using heuristic agent") | raise `AI_CPUS` and `AI_WORKERS` (thread-safe agent!) or use a smaller model; `AI_DEGRADE_QUEUE` sets how early it degrades |
| Server logs "AI failed 5 times in a row, pausing AI requests" | the AI service hangs or crashes: check `docker compose logs wizard_ai` |
| Server logs "… AI requests already waiting" | more simultaneous AI turns than `AI_MAX_INFLIGHT`: raise it to ≥ `MAX_ACTIVE_GAMES` |
| Container restarts, `docker inspect` shows `OOMKilled: true` | raise `SERVER_MEMORY` / `AI_MEMORY` |

## Monitoring

- `curl -s https://game.marvinkrausser.com/wizard/api/health` — load, CPU,
  games, connections, AI state (see above).
- `docker stats` — live CPU and memory per container against the caps.
- Log lines to watch (`docker compose logs -f wizard_server wizard_ai`):

  | Log line | Meaning |
  |----------|---------|
  | `Load level normal -> busy` / `busy -> overloaded` / `-> normal` | load level changes (with lag, CPU, throttling) |
  | `Refusing connection from <ip>: connection limit reached` | connection limits hit |
  | `Closing connection from <ip>: message flood` | a client was cut off |
  | `AI failed N times in a row, pausing AI requests for 30s` | circuit breaker opened |
  | `Playing random move for player N: …` | an AI move fell back to random (reason included) |
  | AI: `Overloaded: N requests in progress` / `Dropping expired request` / `Busy (N waiting): using heuristic agent` | the AI service shedding load |

## Testing it yourself

Automated tests:

- `wizard_server/tests/test_limits.py` — token bucket, limiters, load levels
  and hysteresis, cgroup parsing, AI circuit breaker and in-flight cap.
- `wizard_server/tests/test_protection.py` — floods, connection limits, lobby
  creation rate, forced overload, `MAX_ACTIVE_GAMES`, a game continuing while
  overloaded, `server_status` messages.
- `wizard_ai/tests/test_ai.py` — queue full, expired requests, degradation.

Manual checks:

```bash
# Refusal UI without real load
cd wizard_server && LOAD_FORCE_LEVEL=overloaded AI_SHARED_SECRET=dev uvicorn app.main:app

# Game limit: start one game, then a second lobby's "Start game" is refused
cd wizard_server && MAX_ACTIVE_GAMES=1 AI_SHARED_SECRET=dev uvicorn app.main:app

# Real CPU exhaustion on Linux: cap the server at 15 % of a CPU,
# then run many games without delays (e.g. a small websocket script)
cd wizard_server && AI_MOVE_DELAY=0 TRICK_PAUSE=0 MAX_CONNECTIONS_PER_IP=200 \
  MESSAGE_RATE=1000 MESSAGE_BURST=1000 AI_SHARED_SECRET=dev \
  systemd-run --user --scope -p CPUQuota=15% uvicorn app.main:app
```

## Known limitations

- A running game whose humans all disconnected keeps its `MAX_ACTIVE_GAMES`
  slot until their sessions expire (`SESSION_TTL`, 30 min), because they may
  reconnect. It costs no CPU meanwhile.
- The limits are per server process. That is fine because the server must run
  as a single process anyway (all state is in memory).
- Players behind one IP share `MAX_CONNECTIONS_PER_IP` and `LOBBY_CREATE_RATE`.
- There is no waiting queue: players have to retry when the server is busy.