Add resource limits, abuse protection and overload handling

Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.

- Docker: CPU/memory/pids caps per container (overridable in .env);
  uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
  rate per connection (token bucket, close 1008 on floods), lobby
  creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
  throttling, with hysteresis (normal/busy/overloaded, severe lag
  overloads immediately)
- Admission control: create_lobby and start_game refused while
  overloaded or MAX_ACTIVE_GAMES are running; server_status message
  broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
  requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
  games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
  limits; docs updated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/04-server.md

@@ -26,7 +26,9 @@ flowchart TB
 | `service.py` | `WizardService` | the "controller": handles a browser connection, dispatches messages, manages sessions and cleanup |
 | `sessions.py` | `Session`, `Connection`, `SessionRegistry` | player identity across reconnects; ordered sending |
 | `lobby.py` | `LobbyManager`, `Lobby`, `Seat`, `LobbyError` | lobby membership, host actions, running the game, AI turns, broadcasting |
-| `ai_bridge.py` | `AIBridge`, `AIConnection` | talks to the AI service; random fallback |
+| `ai_bridge.py` | `AIBridge`, `AIConnection` | talks to the AI service; random fallback, in-flight cap, circuit breaker |
+| `limits.py` | `TokenBucket`, `KeyedRateLimiter`, `ConnectionLimiter` | abuse limits (message rate, connections, lobby creation) |
+| `load.py` | `LoadMonitor`, `next_level`, cgroup helpers | detects when the server runs out of compute |
 
 ## Routes (`main.py`)
 
@@ -249,7 +251,102 @@ The AI service keeps a websocket open to `/internal/ai`. `AIBridge`:
   the action is malformed or illegal (logged as warning, except the
   "no service connected" case which is logged at debug level);
 - calls `on_change` when a service connects/disconnects, and the service then
-  re-broadcasts all lobbies so the lobby screen shows whether the AI is online.
+  re-broadcasts all lobbies so the lobby screen shows whether the AI is online;
+- sends `timeout` with every request so the AI service can drop requests the
+  server has already given up on;
+- **in-flight cap**: if `AI_MAX_INFLIGHT` requests are already waiting on a
+  connection, the move is played by the fallback without asking;
+- **circuit breaker**: after `AI_BREAKER_FAILURES` failures in a row (timeouts,
+  errors, illegal answers) the AI is not asked for `AI_BREAKER_COOLDOWN`
+  seconds, so a struggling AI service is not buried under more requests.
+
+## Limits and overload handling
+
+The server protects the host and its players in three layers. The guiding rule
+is: **running games always continue; new work is refused when there is no
+capacity; abuse is rejected immediately.**
+
+```mermaid
+flowchart TD
+    C[new websocket] --> L1{"connection limit<br/>total / per IP"}
+    L1 -- over --> R1["error busy + close 1013"]
+    L1 -- ok --> M[message]
+    M --> L2{"message rate<br/>token bucket"}
+    L2 -- over --> R2["dropped + error rate_limited<br/>(close 1008 when flooding)"]
+    L2 -- ok --> D{"create_lobby /<br/>start_game?"}
+    D -- no --> OK[handled normally]
+    D -- yes --> L3{"accepting_games?<br/>not overloaded and<br/>running games < MAX_ACTIVE_GAMES"}
+    L3 -- no --> R3["error busy"]
+    L3 -- yes --> OK
+```
+
+### 1. Hard limits (Docker and uvicorn)
+
+- `compose.yaml` caps every container (`deploy.resources.limits` + `pids_limit`):
+  server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB by default,
+  overridable in `.env` (`SERVER_CPUS`, `SERVER_MEMORY`, …).
+- uvicorn (`wizard_server/Dockerfile`): `--ws-max-size 65536` (bigger messages
+  close the connection with 1009), `--ws-max-queue 16`,
+  `--limit-concurrency 1000` (HTTP 503 beyond that — a last resort above the
+  app's own connection limit).
+
+### 2. Abuse limits (`limits.py`, applied in `WizardService`)
+
+| Limit | Default | When exceeded |
+|-------|---------|---------------|
+| open connections total / per client IP | 500 / 10 | `error` (code `busy`), then close **1013** "Try Again Later" |
+| messages per connection (token bucket) | 10/s, burst 30 | message dropped, `error` (code `rate_limited`, not repeated for every message); after 50 drops the connection is closed with 1008 |
+| lobbies created per client IP | 10 per minute | `error` (code `rate_limited`) |
+
+The client IP is `websocket.client.host`. Behind Caddy, uvicorn's
+`--proxy-headers` sets it from `X-Forwarded-For` (Caddy does not pass on
+client-supplied values, so it cannot be spoofed).
+
+### 3. Load monitor and admission control (`load.py`)
+
+`LoadMonitor.run()` (started in the app lifespan) samples once per second:
+
+- **event loop lag** — how late the loop wakes up; the most direct "players
+  notice slowness" signal;
+- **CPU** — process CPU time relative to the container's CPU quota (read from
+  the process's own cgroup, `/proc/self/cgroup` → `…/cpu.max`), at most one
+  core because game logic runs on one core;
+- **throttling** — share of CPU periods in which the container hit its limit
+  (`cpu.stat`: `nr_throttled / nr_periods`).
+
+Each sample is classified (`classify`) and `next_level` derives the level with
+hysteresis:
+
+| Level | Entered when | Left when |
+|-------|--------------|-----------|
+| `busy` | one sample with lag ≥ 0.1 s or CPU ≥ 70 % | 10 normal samples in a row |
+| `overloaded` | 3 samples in a row with lag ≥ 0.25 s, CPU ≥ 90 % or throttled ≥ 50 % — or **one** sample with lag ≥ 1 s | 10 samples in a row that are not overloaded |
+
+`busy` is only informational. While **overloaded**, or while
+`MAX_ACTIVE_GAMES` (default 50) games are running, `accepting_games` is false:
+
+| Action | Refused when not accepting games |
+|--------|----------------------------------|
+| `create_lobby` | yes (`error` code `busy`) |
+| `start_game` | yes (`error` code `busy`) |
+| `join_lobby`, moves, `return_to_lobby`, reconnects | never — they are cheap and keep running games alive |
+
+Refusing new games sheds load by itself: finished games are not replaced, so
+the load drops and the server recovers.
+
+Every connected client receives a `server_status` message after `welcome` and
+whenever the level or `accepting_games` changes (the website then disables
+"Create lobby" / "Start game" and shows a notice). `/health` reports `load`,
+`loop_lag`, `cpu`, `throttled`, `accepting_games`, `active_games`,
+`connections` and AI bridge stats, but its `status` **stays "ok"** under load:
+an overloaded server must never be restarted by Docker, which would end all
+running games.
+
+Measured on a server capped at 15 % of a CPU with 30 games played at full
+speed: overloaded after ~3 s, new lobbies refused, running games kept going,
+back to normal ~10 s after the load dropped.
+
+For manual testing, `LOAD_FORCE_LEVEL=overloaded` pins the level.
 
 ## Configuration (`config.py`)
 
@@ -265,27 +362,51 @@ All values are read from environment variables when the module is imported.
 | `MAX_LOBBIES` | 1000 | limit of simultaneously open lobbies |
 | `ALLOWED_ORIGINS` | empty (= all) | comma separated allowed `Origin` headers for `/ws` |
 | `LOG_LEVEL` | INFO | Python logging level |
+| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | 500 / 10 | open websocket connections |
+| `MESSAGE_RATE` / `MESSAGE_BURST` | 10 / 30 | messages per second per connection |
+| `MAX_RATE_VIOLATIONS` | 50 | dropped messages before a flooding connection is closed |
+| `LOBBY_CREATE_RATE` | 10 | lobbies per IP per minute |
+| `MAX_ACTIVE_GAMES` | 50 | games running at the same time |
+| `LOAD_SAMPLE_INTERVAL` | 1.0 s | load sampling period |
+| `LOAD_LAG_BUSY` / `LOAD_LAG_OVERLOAD` / `LOAD_LAG_SEVERE` | 0.1 / 0.25 / 1.0 s | event loop lag thresholds |
+| `LOAD_CPU_BUSY` / `LOAD_CPU_OVERLOAD` | 0.7 / 0.9 | CPU share of the quota |
+| `LOAD_THROTTLE_OVERLOAD` | 0.5 | throttled share of CPU periods |
+| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | 3 / 10 | hysteresis |
+| `LOAD_FORCE_LEVEL` | empty | pin the level (testing) |
+| `AI_MAX_INFLIGHT` | 32 | waiting AI requests per AI connection |
+| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | 5 / 30 s | AI circuit breaker |
 
 ## Housekeeping
 
 `WizardService.cleanup_loop()` runs every 30 s (started in the app lifespan)
-and removes expired sessions (leaving their lobby first).
+and removes expired sessions (leaving their lobby first) and forgets idle
+per-IP lobby-creation buckets.
 
-## Tests (`wizard_server/tests/test_server.py`)
+## Tests (`wizard_server/tests/`)
 
-Uses FastAPI's `TestClient` websockets with zero delays. Covers: health,
-hello requirement, validation errors, the full lobby flow including a complete
-20-round game between two humans and an AI, reconnect by token, leaving
-mid-game (seat becomes AI, host moves), the AI endpoint's secret and proxy
-checks, and a fake AI service answering decide requests.
+Uses FastAPI's `TestClient` websockets with zero delays and a raised message
+rate limit (`helpers.TEST_SETTINGS`).
 
-Note for writing new tests: enter websocket sessions with
-`client.sockets.enter_context(client.websocket_connect(...))` (see the fixture),
-otherwise the test client hangs on shutdown.
+- `test_server.py` — health, hello requirement, validation errors, the full
+  lobby flow including a complete 20-round game between two humans and an AI,
+  reconnect by token, leaving mid-game (seat becomes AI, host moves), the AI
+  endpoint's secret and proxy checks, and a fake AI service.
+- `test_protection.py` — message flood, connections per IP, lobby creation
+  rate, forced overload, `MAX_ACTIVE_GAMES`, a running game continuing while
+  overloaded, `server_status` messages.
+- `test_limits.py` — unit tests of the token bucket, limiters, load levels,
+  cgroup parsing and the AI circuit breaker / in-flight cap.
+
+Fixtures (`conftest.py`): `client` (default test settings) and
+`make_client(**overrides)` for custom settings. Enter websocket sessions with
+`client.sockets.enter_context(client.websocket_connect(...))` (as
+`helpers.connect` does), otherwise the test client hangs on shutdown.
 
 ## Limitations to be aware of
 
 - In-memory state: restarting the server ends all lobbies and games.
 - Must run as a single process (no `--workers`).
 - No spectators: only seated players receive game state.
-- No rate limiting on websocket messages.
+- A running game whose humans all disconnected keeps its `MAX_ACTIVE_GAMES`
+  slot until their sessions expire (`SESSION_TTL`), because they may still
+  reconnect. It costs no CPU while waiting.

docs/05-protocol.md

@@ -93,25 +93,51 @@ contains the **complete** current state.
 - `ai_available` is `false` when no AI service is connected; AI seats then play
   random moves.
 
+#### `server_status`
+
+Sent right after `welcome` and to every connected client whenever the server's
+load level or capacity changes.
+
+```json
+{"type": "server_status", "load": "overloaded", "accepting_games": false}
+```
+
+- `load`: `normal`, `busy` (informational) or `overloaded`.
+- `accepting_games`: `false` while the server refuses `create_lobby` and
+  `start_game` (overloaded, or `MAX_ACTIVE_GAMES` games running). Everything
+  else, including running games, keeps working.
+
 #### `error`
 
 ```json
 {"type": "error", "message": "Only the host can do that"}
+{"type": "error", "message": "The server is busy right now, …", "code": "busy"}
 ```
 
+`code` is present for errors the client may react to:
+
+| `code` | Meaning |
+|--------|---------|
+| `busy` | no capacity: connection limit reached (followed by close 1013), or `create_lobby` / `start_game` refused because the server does not accept new games |
+| `rate_limited` | too many messages on this connection, or too many lobbies created from this IP |
+
 Sent only to the client that caused it. Typical messages: `Send 'hello' first`,
 `Invalid message: …` (followed by the pydantic reason),
 `Lobby ABC123 does not exist`, `The game in this lobby has already started`,
 `This lobby is full`, `Only the host can do that`,
 `Wizard needs 3 to 6 players`, `Illegal action Bid(value=2) for player 0`,
 `It is not player 1's turn`, `You were removed from the lobby`,
-`This session was opened in another window`.
+`This session was opened in another window`, `Too many messages, slow down.`,
+`You are creating lobbies too quickly, please wait a moment.`
 
 ### Close codes
 
 | Code | Meaning | Frontend reaction |
 |------|---------|-------------------|
 | (handshake rejected) | origin not in `ALLOWED_ORIGINS`; the server closes with 1008 before accepting, so the browser sees a failed connection | reconnects with backoff |
+| 1008 | message flood (too many dropped messages) | reconnects with backoff |
+| 1009 | message larger than 64 KB (uvicorn `--ws-max-size`) | reconnects with backoff |
+| 1013 | "Try Again Later": connection limit reached (preceded by an `error` with code `busy`) | banner "The server is full", retries slowly (15 s, 30 s, 60 s …) |
 | 4000 | the session was opened in another tab | stops reconnecting, shows a banner |
 | other | network problem, server restart | reconnects (1 s, 2 s, 4 s … max 10 s) |
 
@@ -153,9 +179,9 @@ headers (see "Routes" in [the server chapter](04-server.md)).
 | Direction | Message | Meaning |
 |-----------|---------|---------|
 | AI → server | `{"type":"hello","agent":"heuristic"}` | informational, logged |
-| server → AI | `{"type":"decide","request_id":"<hex>","observation":{…}}` | choose an action for `observation.player` |
-| AI → server | `{"type":"action","request_id":"<same>","action":{"type":"play","card":17}}` | the chosen action |
-| AI → server | `{"type":"error","request_id":"<same>","message":"…"}` | the AI failed; the server plays a random move |
+| server → AI | `{"type":"decide","request_id":"<hex>","timeout":5.0,"observation":{…}}` | choose an action for `observation.player`; the server waits at most `timeout` seconds |
+| AI → server | `{"type":"action","request_id":"<same>","action":{"type":"play","card":17},"agent":"heuristic"}` | the chosen action; `agent` names the agent that decided (logged) |
+| AI → server | `{"type":"error","request_id":"<same>","message":"…"}` | the AI failed or refused (`overloaded`, `expired`); the server plays a random move |
 
 Rules:
 
@@ -163,6 +189,11 @@ Rules:
   (matched by `request_id`) and may handle several concurrently.
 - The answer must be one of `observation.legal_actions`; otherwise the server
   plays a random legal move.
-- No answer within `AI_DECISION_TIMEOUT` (5 s) → random move.
+- No answer within `AI_DECISION_TIMEOUT` (5 s) → random move. The AI service
+  drops requests whose `timeout` passed before it could start them (`expired`).
+- The server stops asking an AI connection that already has `AI_MAX_INFLIGHT`
+  unanswered requests, and pauses the AI after repeated failures (circuit
+  breaker); see "Limits and overload handling" in
+  [the server chapter](04-server.md).
 - Several AI service instances may connect; the server distributes requests
   round robin.

docs/06-frontend.md

@@ -53,7 +53,8 @@ The only stateful piece of the app.
 
 | Value | Meaning |
 |-------|---------|
-| `status` | `connecting` \| `open` \| `closed` (will retry) \| `replaced` (opened in another tab, no retry) |
+| `status` | `connecting` \| `open` \| `closed` (will retry) \| `replaced` (opened in another tab, no retry) \| `busy` (server full, retries slowly) |
+| `serverStatus` | last `server_status` message: `{ load, accepting_games }` |
 | `playerId` | from `welcome` |
 | `lobby`, `game` | exactly what the last `state` message contained |
 | `completedTrick` | the last `trick_completed` event, kept for 2.2 s so the trick stays visible |
@@ -73,6 +74,10 @@ The only stateful piece of the app.
   second tab is a different player.
 - On close it reconnects with exponential backoff (1, 2, 4, 8, 10, 10 … s),
   except after close code 4000.
+- If the server refuses the connection because it is full (an `error` with
+  code `busy` before `welcome`, then close code 1013), the status becomes
+  `busy` and it retries slowly (15 s, 30 s, 60 s, …) so waiting clients don't
+  add load.
 - `send()` shows a toast if the socket is not open.
 
 ## Screen selection (`App.tsx`)
@@ -96,6 +101,8 @@ shows the connection status when it is not `open`.
 - Code input + "Join" → `actions.joinLobby(code, name)`. The code is prefilled
   from `?lobby=` so invite links work.
 - Buttons are disabled while disconnected or without a name.
+- While `serverStatus.accepting_games` is false, "Create lobby" is disabled and
+  a notice explains that the server is busy (joining still works).
 
 ### LobbyScreen
 
@@ -107,6 +114,8 @@ shows the connection status when it is not `open`.
   seats are filled). Others see the options read-only and "Waiting for the
   host…".
 - Shows a warning if AI players are present but the AI service is offline.
+- While `serverStatus.accepting_games` is false, "Start game" shows
+  "Server busy" (disabled) with a notice; the lobby stays open.
 - Explains the fixed bidding rule (bids may not add up to the cards in hand).
 
 ### GameScreen (`screens/GameScreen/`)

docs/07-ai.md

@@ -194,8 +194,11 @@ registered with `@register_policy`, and calls its `load`.
    the container (`compose.yaml`); restart `wizard_ai`. With `AGENT=model`
    (default) it is loaded automatically.
 
-Model size matters: the service answers every AI move in a worker thread and
-the server waits at most `AI_DECISION_TIMEOUT` (5 s).
+Model size matters: the service answers every AI move in a worker thread, the
+container is capped at `AI_CPUS` (1 CPU by default) and the server waits at
+most `AI_DECISION_TIMEOUT` (5 s). If the model is too slow the service
+automatically answers with the heuristic instead (see below), so measure the
+decision time before deploying.
 
 ## Training (`training/`)
 
@@ -244,14 +247,28 @@ count as a win for everyone tied). Seeded for fair comparisons.
 
 ## Production service (`service/client.py`)
 
-`AIService(server_url, agent, secret).run_forever()`:
+`AIService(server_url, agent, secret, fallback=…, workers=…, max_queue=…, degrade_queue=…).run_forever()`:
 
 - connects to `SERVER_URL` with `Authorization: Bearer <AI_SHARED_SECRET>`,
   sends `hello`, then answers `decide` requests (protocol in
   [chapter 5](05-protocol.md#ai-protocol));
-- each request runs `agent.act` in a worker thread (`asyncio.to_thread`) behind
-  a lock, so a slow model does not block the websocket and non-thread-safe
-  agents stay consistent;
+- each request runs `agent.act` in a worker thread (`asyncio.to_thread`), so a
+  slow model does not block the websocket;
+- **bounded work** (`handle_request`):
+  - at most `AI_WORKERS` (default 1) decisions are computed at the same time —
+    agents must be thread safe if you raise it;
+  - at most `AI_MAX_QUEUE` (64) requests may be in progress; more are answered
+    with `error` "overloaded" immediately;
+  - a request whose server `timeout` passed while it waited for a worker is
+    answered with `error` "expired" without computing it;
+- **degradation**: when `AI_DEGRADE_QUEUE` (8) or more other requests are
+  waiting, or the primary agent's average decision time is more than half of
+  the remaining time, the cheap fallback agent (`HeuristicAgent`) decides
+  instead. `serve` sets the heuristic as fallback automatically unless the
+  primary agent already is the heuristic. The reply's `agent` field says which
+  one decided;
+- `OMP_NUM_THREADS` / `MKL_NUM_THREADS` are set to `AI_WORKERS` in
+  `compose.yaml`, so numeric libraries don't start a thread per host core;
 - errors while deciding are reported back as `error` (the server then plays a
   random move) instead of crashing;
 - reconnects forever with exponential backoff (1 s … 30 s); logs a hint if the
@@ -271,10 +288,13 @@ Environment variables (`config.py`): `SERVER_URL`
 (default `ws://localhost:8000/internal/ai`; in Docker
 `ws://wizard_server:8083/internal/ai`), `AI_SHARED_SECRET`, `AGENT`
 (default `model`), `MODEL_PATH` (default `models/latest`, in Docker
-`/srv/models/latest`), `LOG_LEVEL`.
+`/srv/models/latest`), `LOG_LEVEL`, `AI_WORKERS` (1), `AI_MAX_QUEUE` (64),
+`AI_DEGRADE_QUEUE` (8).
 
 ## Tests (`wizard_ai/tests/test_ai.py`)
 
 Feature length and masks for 3–6 players, action-space round trip, all agents
 play legal full games, env rewards add up to scores, trainer runs and writes a
-loadable checkpoint, `make_agent` fallback, heuristic beats random.
+loadable checkpoint, `make_agent` fallback, heuristic beats random, and the
+service limits (queue full → overloaded, expired requests, degradation to the
+heuristic when busy or when the model is too slow).

docs/08-operations.md

@@ -44,8 +44,16 @@ not read by anything.
 | `ALLOWED_ORIGINS` | server | no | e.g. `https://game.marvinkrausser.com` |
 | `AI_AGENT` | AI | no (default `model`) | `model`, `heuristic` or `random` |
 | `LOG_LEVEL` | server + AI | no | `INFO`, `DEBUG`, … |
+| `SERVER_CPUS` / `SERVER_MEMORY` | compose | no (1.0 / 512M) | hard caps of the server container |
+| `AI_CPUS` / `AI_MEMORY` | compose | no (1.0 / 1G) | hard caps of the AI container |
+| `WEBSITE_CPUS` / `WEBSITE_MEMORY` | compose | no (0.25 / 64M) | hard caps of the website container |
+| `MAX_ACTIVE_GAMES` | server | no (50) | games running at the same time; more starts are refused ("server busy") |
+| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | no (500 / 10) | open websocket connections |
+| `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | no (1 / 64 / 8) | parallel decisions, waiting requests before refusing, queue depth from which the heuristic is used |
 
-Further server settings (`AI_MOVE_DELAY`, `TRICK_PAUSE`, …, see
+How the limits work together is described in "Limits and overload handling"
+in [the server chapter](04-server.md). Further server settings
+(`AI_MOVE_DELAY`, `TRICK_PAUSE`, load thresholds, …, see
 "Configuration" in [the server chapter](04-server.md)) can be added to
 the `environment:` section of `wizard_server` in `compose.yaml`.
 
@@ -147,8 +155,8 @@ cd wizard_website && npm run lint && npm run build
 | Suite | Count | Covers |
 |-------|-------|--------|
 | game engine | 52 | rules, full random games for 3–6 players, invariants, JSON |
-| server | 9 | websocket flows incl. a complete game, reconnect, AI endpoint security |
-| AI | 13 | encoding, agents, env, trainer, checkpoints |
+| server | 32 | websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels |
+| AI | 17 | encoding, agents, env, trainer, checkpoints, service limits |
 
 ## Troubleshooting
 
@@ -161,6 +169,10 @@ cd wizard_website && npm run lint && npm run build
 | AI players only make weird/random moves | Same as above (random fallback), or the AI answers too slowly (`AI_DECISION_TIMEOUT`), see server logs "Playing random move for player …". |
 | Game waits and nothing happens | It is a disconnected human's turn (seat shows "offline"). The game waits until they reconnect or their session expires (`SESSION_TTL`, then an AI takes over). |
 | All lobbies gone | The server restarted — state is in memory only. |
+| Start page / lobby says "The server is busy", Create/Start disabled | The server is overloaded or `MAX_ACTIVE_GAMES` games are running. Check `https://…/wizard/api/health` (`load`, `cpu`, `throttled`, `active_games`) and `docker stats`. Running games are not affected. If it happens often, raise `SERVER_CPUS` or `MAX_ACTIVE_GAMES`. |
+| Banner "The server is full right now" | Connection limit reached (`MAX_CONNECTIONS`, or `MAX_CONNECTIONS_PER_IP` — e.g. many tabs behind one NAT). The page retries on its own every 15–60 s. |
+| Error toast "Too many messages, slow down." | A client exceeded the message rate limit (10/s). Normal play never gets near it; a script or broken client might. |
+| AI plays noticeably worse under load | The AI service switched to the heuristic because it was busy (`AI_DEGRADE_QUEUE`) or the model was too slow; see `docker compose logs wizard_ai` ("Busy … using heuristic agent"). Raise `AI_CPUS`/`AI_WORKERS` or use a smaller model. |
 | Tests pass but behaviour does not change after editing `wizard_game` | A non-editable copy of `wizard_game` is installed in your virtualenv and shadows the source. `pip uninstall wizard-game && pip install -e wizard_server/game_logic`. Check with `python -c "import wizard_game; print(wizard_game.__file__)"`. |
 | Server tests hang | A test opened a websocket without a context manager; use `client.sockets.enter_context(client.websocket_connect(...))`. |
 
@@ -190,6 +202,17 @@ cd wizard_website && npm run lint && npm run build
    `actions`.
 4. Add a test in `wizard_server/tests/test_server.py`.
 
+### Check resource usage
+
+```bash
+docker stats wizard-wizard_server-1 wizard-wizard_ai-1 wizard-wizard_website-1
+curl -s https://game.marvinkrausser.com/wizard/api/health
+```
+
+To test the overload behaviour without real load, start the server with
+`LOAD_FORCE_LEVEL=overloaded` (new games are refused, the UI shows the busy
+notice).
+
 ### Change how fast AI players move
 
 Set `AI_MOVE_DELAY` (per move) and `TRICK_PAUSE` (after a trick) for