Document limits, load control and performance

- New docs/09-limits-and-performance.md: principles, all protection
  layers, what players see, configuration reference, measurements,
  tuning guide, monitoring and how to test
- Server chapter summarises the limits and links to chapter 9; circuit
  breaker description and AI_MAX_INFLIGHT default corrected
- Architecture: new files in the layout, design decisions for load
  handling; docs index lists chapter 9
- Protocol, AI and operations chapters link to chapter 9; test counts
- Root and component READMEs describe the limits and settings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/04-server.md

@@ -256,97 +256,35 @@ The AI service keeps a websocket open to `/internal/ai`. `AIBridge`:
   server has already given up on;
 - **in-flight cap**: if `AI_MAX_INFLIGHT` requests are already waiting on a
   connection, the move is played by the fallback without asking;
-- **circuit breaker**: after `AI_BREAKER_FAILURES` failures in a row (timeouts,
-  errors, illegal answers) the AI is not asked for `AI_BREAKER_COOLDOWN`
-  seconds, so a struggling AI service is not buried under more requests.
+- **circuit breaker**: after `AI_BREAKER_FAILURES` real failures in a row
+  (timeouts, lost connection, malformed/illegal answers, unexpected errors) the
+  AI is not asked for `AI_BREAKER_COOLDOWN` seconds, so games don't wait
+  `AI_DECISION_TIMEOUT` for every move of a hanging AI. Load shedding (the
+  in-flight cap, AI answers `overloaded` / `expired`) does not count.
 
 ## Limits and overload handling
 
-The server protects the host and its players in three layers. The guiding rule
-is: **running games always continue; new work is refused when there is no
-capacity; abuse is rejected immediately.**
+The complete description — all layers, thresholds, what players see, tuning,
+monitoring and measurements — is in
+[chapter 9: Limits, load control and performance](09-limits-and-performance.md).
+In short, for the server:
 
-```mermaid
-flowchart TD
-    C[new websocket] --> L1{"connection limit<br/>total / per IP"}
-    L1 -- over --> R1["error busy + close 1013"]
-    L1 -- ok --> M[message]
-    M --> L2{"message rate<br/>token bucket"}
-    L2 -- over --> R2["dropped + error rate_limited<br/>(close 1008 when flooding)"]
-    L2 -- ok --> D{"create_lobby /<br/>start_game?"}
-    D -- no --> OK[handled normally]
-    D -- yes --> L3{"accepting_games?<br/>not overloaded and<br/>running games < MAX_ACTIVE_GAMES"}
-    L3 -- no --> R3["error busy"]
-    L3 -- yes --> OK
-```
-
-### 1. Hard limits (Docker and uvicorn)
-
-- `compose.yaml` caps every container (`deploy.resources.limits` + `pids_limit`):
-  server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB by default,
-  overridable in `.env` (`SERVER_CPUS`, `SERVER_MEMORY`, …).
-- uvicorn (`wizard_server/Dockerfile`): `--ws-max-size 65536` (bigger messages
-  close the connection with 1009), `--ws-max-queue 16`,
-  `--limit-concurrency 1000` (HTTP 503 beyond that — a last resort above the
-  app's own connection limit).
-
-### 2. Abuse limits (`limits.py`, applied in `WizardService`)
-
-| Limit | Default | When exceeded |
-|-------|---------|---------------|
-| open connections total / per client IP | 500 / 10 | `error` (code `busy`), then close **1013** "Try Again Later" |
-| messages per connection (token bucket) | 10/s, burst 30 | message dropped, `error` (code `rate_limited`, not repeated for every message); after 50 drops the connection is closed with 1008 |
-| lobbies created per client IP | 10 per minute | `error` (code `rate_limited`) |
-
-The client IP is `websocket.client.host`. Behind Caddy, uvicorn's
-`--proxy-headers` sets it from `X-Forwarded-For` (Caddy does not pass on
-client-supplied values, so it cannot be spoofed).
-
-### 3. Load monitor and admission control (`load.py`)
-
-`LoadMonitor.run()` (started in the app lifespan) samples once per second:
-
-- **event loop lag** — how late the loop wakes up; the most direct "players
-  notice slowness" signal;
-- **CPU** — process CPU time relative to the container's CPU quota (read from
-  the process's own cgroup, `/proc/self/cgroup` → `…/cpu.max`), at most one
-  core because game logic runs on one core;
-- **throttling** — share of CPU periods in which the container hit its limit
-  (`cpu.stat`: `nr_throttled / nr_periods`).
-
-Each sample is classified (`classify`) and `next_level` derives the level with
-hysteresis:
-
-| Level | Entered when | Left when |
-|-------|--------------|-----------|
-| `busy` | one sample with lag ≥ 0.1 s or CPU ≥ 70 % | 10 normal samples in a row |
-| `overloaded` | 3 samples in a row with lag ≥ 0.25 s, CPU ≥ 90 % or throttled ≥ 50 % — or **one** sample with lag ≥ 1 s | 10 samples in a row that are not overloaded |
-
-`busy` is only informational. While **overloaded**, or while
-`MAX_ACTIVE_GAMES` (default 50) games are running, `accepting_games` is false:
-
-| Action | Refused when not accepting games |
-|--------|----------------------------------|
-| `create_lobby` | yes (`error` code `busy`) |
-| `start_game` | yes (`error` code `busy`) |
-| `join_lobby`, moves, `return_to_lobby`, reconnects | never — they are cheap and keep running games alive |
-
-Refusing new games sheds load by itself: finished games are not replaced, so
-the load drops and the server recovers.
-
-Every connected client receives a `server_status` message after `welcome` and
-whenever the level or `accepting_games` changes (the website then disables
-"Create lobby" / "Start game" and shows a notice). `/health` reports `load`,
-`loop_lag`, `cpu`, `throttled`, `accepting_games`, `active_games`,
-`connections` and AI bridge stats, but its `status` **stays "ok"** under load:
-an overloaded server must never be restarted by Docker, which would end all
-running games.
-
-Measured on a server capped at 15 % of a CPU with 30 games played at full
-speed: overloaded after ~3 s, new lobbies refused, running games kept going,
-back to normal ~10 s after the load dropped.
-
-For manual testing, `LOAD_FORCE_LEVEL=overloaded` pins the level.
+- **Rule**: running games always continue; new lobbies/games are refused when
+  there is no capacity; abuse is rejected immediately.
+- **Abuse limits** (`limits.py`, applied in `WizardService.handle_client`):
+  connections in total and per IP (refused with `error` code `busy` + close
+  1013), a token bucket per connection for messages (`rate_limited`, close
+  1008 on floods), lobby creation per IP.
+- **Load monitor** (`load.py`, `LoadMonitor.run()` in the app lifespan):
+  samples event loop lag, CPU against the container quota, and cgroup
+  throttling every second; levels `normal` / `busy` / `overloaded` with
+  hysteresis.
+- **Admission control** (`WizardService.accepting_games`): `create_lobby` and
+  `start_game` are refused while overloaded or while `MAX_ACTIVE_GAMES` games
+  run; everything else keeps working. Changes are broadcast to all clients as
+  `server_status`.
+- `/health` reports the load but its `status` stays "ok", so Docker never
+  restarts an overloaded server (which would end all games).
 
 ## Configuration (`config.py`)
 
@@ -373,7 +311,7 @@ All values are read from environment variables when the module is imported.
 | `LOAD_THROTTLE_OVERLOAD` | 0.5 | throttled share of CPU periods |
 | `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | 3 / 10 | hysteresis |
 | `LOAD_FORCE_LEVEL` | empty | pin the level (testing) |
-| `AI_MAX_INFLIGHT` | 32 | waiting AI requests per AI connection |
+| `AI_MAX_INFLIGHT` | 64 | waiting AI requests per AI connection (keep ≥ `MAX_ACTIVE_GAMES`) |
 | `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | 5 / 30 s | AI circuit breaker |
 
 ## Housekeeping