diff --git a/README.md b/README.md index 256fdeb..ae948d7 100644 --- a/README.md +++ b/README.md @@ -33,10 +33,25 @@ The `Observation` / action JSON format is defined once in `wizard_game` and shar frontend, the server and the AI, so AI models see exactly the same data in training and in production. +## Limits and load control + +Wizard shares its host with other services, so its resource use is bounded in layers +(details: [`docs/09-limits-and-performance.md`](docs/09-limits-and-performance.md)): + +- **Docker caps** per container (server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB), + set in `.env`. +- **Abuse limits**: connections per IP and in total, messages per second per connection, + lobbies per IP per minute. +- **Overload handling**: the server watches event loop lag, CPU and cgroup throttling. While it + is overloaded, or `MAX_ACTIVE_GAMES` games are running, new lobbies and game starts are refused + ("server busy" in the UI). Running games always continue. +- **AI cost bounds**: bounded queue and workers in the AI service, which falls back to the cheap + heuristic player when busy; circuit breaker and in-flight cap in the server. + ## Deployment ```bash -cp .env.example .env # set AI_SHARED_SECRET +cp .env.example .env # set AI_SHARED_SECRET (limits optional) docker network create wizard_website # if they don't exist yet docker network create wizard_api docker compose up -d --build diff --git a/docs/01-architecture.md b/docs/01-architecture.md index 7d12981..21483e3 100644 --- a/docs/01-architecture.md +++ b/docs/01-architecture.md @@ -5,7 +5,7 @@ | Component | Folder | Language / stack | Runs as | Responsibility | |-----------|--------|------------------|---------|----------------| | Website | `wizard_website/` | React 19, TypeScript, Vite, CSS modules | static files served by nginx (container port 8082) | UI: start page, lobby, game table | -| Game server | `wizard_server/app/` | Python 3.13, FastAPI, uvicorn | one process (container port 8083) | sessions, lobbies, running games, asking the AI for moves | +| Game server | `wizard_server/app/` | Python 3.13, FastAPI, uvicorn | one process (container port 8083) | sessions, lobbies, running games, asking the AI for moves, limits and load control | | Game engine | `wizard_server/game_logic/` (package `wizard_game`) | pure Python, no dependencies | library, installed into server and AI images | the rules of Wizard | | AI service | `wizard_ai/` | Python, `websockets` | one process, no listening port | chooses moves for AI seats; training tools | @@ -89,8 +89,8 @@ Important properties of this flow: ``` . -├── compose.yaml # all three containers + networks -├── .env.example # template for .env (AI_SHARED_SECRET, …) +├── compose.yaml # all three containers, networks, CPU/memory caps +├── .env.example # template for .env (secret, resource limits, capacity) ├── README.md # short project readme ├── docs/ # this documentation ├── instructions.txt # original task description @@ -107,8 +107,11 @@ Important properties of this flow: │ │ ├── service.py # WizardService: dispatches messages, sessions, cleanup │ │ ├── sessions.py # Session, Connection (ordered send queue), SessionRegistry │ │ ├── lobby.py # Lobby, Seat, LobbyManager, AI turn loop -│ │ └── ai_bridge.py # request/response with the AI service + fallback -│ ├── tests/test_server.py # websocket-level tests of the server +│ │ ├── ai_bridge.py # request/response with the AI service, fallback, circuit breaker +│ │ ├── limits.py # TokenBucket, rate and connection limiters +│ │ └── load.py # LoadMonitor: detects overload (lag, CPU, cgroup throttling) +│ ├── tests/ # conftest.py/helpers.py (fixtures), test_server.py, +│ │ # test_protection.py (limits/overload), test_limits.py (units) │ └── game_logic/ # installable package "wizard-game" │ ├── pyproject.toml │ ├── wizard_game/ @@ -130,7 +133,7 @@ Important properties of this flow: │ │ ├── agents/ # Agent interface, random, heuristic, model agent │ │ ├── model/ # encoding.py (features/actions), policy.py (model interface) │ │ ├── training/ # env.py, self_play.py, evaluate.py -│ │ └── service/client.py # production websocket client +│ │ └── service/client.py # production websocket client (workers, queue, degradation) │ └── tests/test_ai.py │ └── wizard_website/ @@ -158,3 +161,6 @@ Important properties of this flow: | Session token in `sessionStorage` | Survives page reloads (reconnect to the same seat) but is per tab, so you can test several players in several tabs. | | Per-connection send queue | Websocket messages are always sent in order and never concurrently (see `sessions.Connection`). | | Relative asset paths + relative API URL in the frontend | The app works under any path prefix (`/wizard/website/` in production, `/` in development) without rebuilding. | +| Refuse new games under load, never interrupt running ones | Players in a game are never punished for load; creating/starting games is what adds load, so that is what gets refused ("server busy"). Refusing sheds load by itself. See [chapter 9](09-limits-and-performance.md). | +| Hard Docker caps plus software limits | The caps guarantee the host is protected even if a software limit fails; the software limits make overload visible and graceful instead of just slow. | +| Health check stays "ok" under load | Docker must never restart an overloaded server — that would end every running game. | diff --git a/docs/04-server.md b/docs/04-server.md index d7f082f..7b09546 100644 --- a/docs/04-server.md +++ b/docs/04-server.md @@ -256,97 +256,35 @@ The AI service keeps a websocket open to `/internal/ai`. `AIBridge`: server has already given up on; - **in-flight cap**: if `AI_MAX_INFLIGHT` requests are already waiting on a connection, the move is played by the fallback without asking; -- **circuit breaker**: after `AI_BREAKER_FAILURES` failures in a row (timeouts, - errors, illegal answers) the AI is not asked for `AI_BREAKER_COOLDOWN` - seconds, so a struggling AI service is not buried under more requests. +- **circuit breaker**: after `AI_BREAKER_FAILURES` real failures in a row + (timeouts, lost connection, malformed/illegal answers, unexpected errors) the + AI is not asked for `AI_BREAKER_COOLDOWN` seconds, so games don't wait + `AI_DECISION_TIMEOUT` for every move of a hanging AI. Load shedding (the + in-flight cap, AI answers `overloaded` / `expired`) does not count. ## Limits and overload handling -The server protects the host and its players in three layers. The guiding rule -is: **running games always continue; new work is refused when there is no -capacity; abuse is rejected immediately.** +The complete description — all layers, thresholds, what players see, tuning, +monitoring and measurements — is in +[chapter 9: Limits, load control and performance](09-limits-and-performance.md). +In short, for the server: -```mermaid -flowchart TD - C[new websocket] --> L1{"connection limit
total / per IP"} - L1 -- over --> R1["error busy + close 1013"] - L1 -- ok --> M[message] - M --> L2{"message rate
token bucket"} - L2 -- over --> R2["dropped + error rate_limited
(close 1008 when flooding)"] - L2 -- ok --> D{"create_lobby /
start_game?"} - D -- no --> OK[handled normally] - D -- yes --> L3{"accepting_games?
not overloaded and
running games < MAX_ACTIVE_GAMES"} - L3 -- no --> R3["error busy"] - L3 -- yes --> OK -``` - -### 1. Hard limits (Docker and uvicorn) - -- `compose.yaml` caps every container (`deploy.resources.limits` + `pids_limit`): - server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB by default, - overridable in `.env` (`SERVER_CPUS`, `SERVER_MEMORY`, …). -- uvicorn (`wizard_server/Dockerfile`): `--ws-max-size 65536` (bigger messages - close the connection with 1009), `--ws-max-queue 16`, - `--limit-concurrency 1000` (HTTP 503 beyond that — a last resort above the - app's own connection limit). - -### 2. Abuse limits (`limits.py`, applied in `WizardService`) - -| Limit | Default | When exceeded | -|-------|---------|---------------| -| open connections total / per client IP | 500 / 10 | `error` (code `busy`), then close **1013** "Try Again Later" | -| messages per connection (token bucket) | 10/s, burst 30 | message dropped, `error` (code `rate_limited`, not repeated for every message); after 50 drops the connection is closed with 1008 | -| lobbies created per client IP | 10 per minute | `error` (code `rate_limited`) | - -The client IP is `websocket.client.host`. Behind Caddy, uvicorn's -`--proxy-headers` sets it from `X-Forwarded-For` (Caddy does not pass on -client-supplied values, so it cannot be spoofed). - -### 3. Load monitor and admission control (`load.py`) - -`LoadMonitor.run()` (started in the app lifespan) samples once per second: - -- **event loop lag** — how late the loop wakes up; the most direct "players - notice slowness" signal; -- **CPU** — process CPU time relative to the container's CPU quota (read from - the process's own cgroup, `/proc/self/cgroup` → `…/cpu.max`), at most one - core because game logic runs on one core; -- **throttling** — share of CPU periods in which the container hit its limit - (`cpu.stat`: `nr_throttled / nr_periods`). - -Each sample is classified (`classify`) and `next_level` derives the level with -hysteresis: - -| Level | Entered when | Left when | -|-------|--------------|-----------| -| `busy` | one sample with lag ≥ 0.1 s or CPU ≥ 70 % | 10 normal samples in a row | -| `overloaded` | 3 samples in a row with lag ≥ 0.25 s, CPU ≥ 90 % or throttled ≥ 50 % — or **one** sample with lag ≥ 1 s | 10 samples in a row that are not overloaded | - -`busy` is only informational. While **overloaded**, or while -`MAX_ACTIVE_GAMES` (default 50) games are running, `accepting_games` is false: - -| Action | Refused when not accepting games | -|--------|----------------------------------| -| `create_lobby` | yes (`error` code `busy`) | -| `start_game` | yes (`error` code `busy`) | -| `join_lobby`, moves, `return_to_lobby`, reconnects | never — they are cheap and keep running games alive | - -Refusing new games sheds load by itself: finished games are not replaced, so -the load drops and the server recovers. - -Every connected client receives a `server_status` message after `welcome` and -whenever the level or `accepting_games` changes (the website then disables -"Create lobby" / "Start game" and shows a notice). `/health` reports `load`, -`loop_lag`, `cpu`, `throttled`, `accepting_games`, `active_games`, -`connections` and AI bridge stats, but its `status` **stays "ok"** under load: -an overloaded server must never be restarted by Docker, which would end all -running games. - -Measured on a server capped at 15 % of a CPU with 30 games played at full -speed: overloaded after ~3 s, new lobbies refused, running games kept going, -back to normal ~10 s after the load dropped. - -For manual testing, `LOAD_FORCE_LEVEL=overloaded` pins the level. +- **Rule**: running games always continue; new lobbies/games are refused when + there is no capacity; abuse is rejected immediately. +- **Abuse limits** (`limits.py`, applied in `WizardService.handle_client`): + connections in total and per IP (refused with `error` code `busy` + close + 1013), a token bucket per connection for messages (`rate_limited`, close + 1008 on floods), lobby creation per IP. +- **Load monitor** (`load.py`, `LoadMonitor.run()` in the app lifespan): + samples event loop lag, CPU against the container quota, and cgroup + throttling every second; levels `normal` / `busy` / `overloaded` with + hysteresis. +- **Admission control** (`WizardService.accepting_games`): `create_lobby` and + `start_game` are refused while overloaded or while `MAX_ACTIVE_GAMES` games + run; everything else keeps working. Changes are broadcast to all clients as + `server_status`. +- `/health` reports the load but its `status` stays "ok", so Docker never + restarts an overloaded server (which would end all games). ## Configuration (`config.py`) @@ -373,7 +311,7 @@ All values are read from environment variables when the module is imported. | `LOAD_THROTTLE_OVERLOAD` | 0.5 | throttled share of CPU periods | | `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | 3 / 10 | hysteresis | | `LOAD_FORCE_LEVEL` | empty | pin the level (testing) | -| `AI_MAX_INFLIGHT` | 32 | waiting AI requests per AI connection | +| `AI_MAX_INFLIGHT` | 64 | waiting AI requests per AI connection (keep ≥ `MAX_ACTIVE_GAMES`) | | `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | 5 / 30 s | AI circuit breaker | ## Housekeeping diff --git a/docs/05-protocol.md b/docs/05-protocol.md index 3618d33..f3ec9a3 100644 --- a/docs/05-protocol.md +++ b/docs/05-protocol.md @@ -192,8 +192,9 @@ Rules: - No answer within `AI_DECISION_TIMEOUT` (5 s) → random move. The AI service drops requests whose `timeout` passed before it could start them (`expired`). - The server stops asking an AI connection that already has `AI_MAX_INFLIGHT` - unanswered requests, and pauses the AI after repeated failures (circuit - breaker); see "Limits and overload handling" in - [the server chapter](04-server.md). + unanswered requests, and pauses the AI after repeated real failures (circuit + breaker). `overloaded` / `expired` answers are load shedding, not failures, + and do not trip the breaker. See + "Layer 4" in [chapter 9](09-limits-and-performance.md). - Several AI service instances may connect; the server distributes requests round robin. diff --git a/docs/07-ai.md b/docs/07-ai.md index ad56cf3..d409016 100644 --- a/docs/07-ai.md +++ b/docs/07-ai.md @@ -268,7 +268,9 @@ count as a win for everyone tied). Seeded for fair comparisons. primary agent already is the heuristic. The reply's `agent` field says which one decided; - `OMP_NUM_THREADS` / `MKL_NUM_THREADS` are set to `AI_WORKERS` in - `compose.yaml`, so numeric libraries don't start a thread per host core; + `compose.yaml`, so numeric libraries don't start a thread per host core + (the whole picture, including the server side, is in + [chapter 9](09-limits-and-performance.md)); - errors while deciding are reported back as `error` (the server then plays a random move) instead of crashing; - reconnects forever with exponential backoff (1 s … 30 s); logs a hint if the diff --git a/docs/08-operations.md b/docs/08-operations.md index eb81a43..47e20c1 100644 --- a/docs/08-operations.md +++ b/docs/08-operations.md @@ -51,8 +51,8 @@ not read by anything. | `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | no (500 / 10) | open websocket connections | | `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | no (1 / 64 / 8) | parallel decisions, waiting requests before refusing, queue depth from which the heuristic is used | -How the limits work together is described in "Limits and overload handling" -in [the server chapter](04-server.md). Further server settings +How the limits work together, the full list of settings, measurements and a +tuning guide are in [chapter 9](09-limits-and-performance.md). Further server settings (`AI_MOVE_DELAY`, `TRICK_PAUSE`, load thresholds, …, see "Configuration" in [the server chapter](04-server.md)) can be added to the `environment:` section of `wizard_server` in `compose.yaml`. @@ -155,7 +155,7 @@ cd wizard_website && npm run lint && npm run build | Suite | Count | Covers | |-------|-------|--------| | game engine | 52 | rules, full random games for 3–6 players, invariants, JSON | -| server | 32 | websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels | +| server | 33 | websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels | | AI | 17 | encoding, agents, env, trainer, checkpoints, service limits | ## Troubleshooting diff --git a/docs/09-limits-and-performance.md b/docs/09-limits-and-performance.md new file mode 100644 index 0000000..8d85db1 --- /dev/null +++ b/docs/09-limits-and-performance.md @@ -0,0 +1,373 @@ +# 9. Limits, load control and performance + +This chapter explains how Wizard keeps its resource use bounded, how it +protects itself against abuse, and what happens when it runs out of compute. +It covers all components, because the protection works in layers from Docker +down to the browser. + +## Principles + +1. **Running games always continue.** A game in progress is never stopped, + slowed down on purpose or refused a move because of load. +2. **New work is refused when there is no capacity.** Creating lobbies and + starting games is what adds load, so that is what gets refused — with a + clear "server busy" message, not a timeout. (A waiting queue was considered + and rejected as not worth the complexity.) +3. **Abuse is rejected immediately.** Floods and connection storms are cut off + at the door, never queued. +4. **The host is protected no matter what.** Hard Docker caps guarantee that + Wizard can never take CPU or memory from the other services on the machine, + even if every software limit fails. +5. **Degrade, don't fail.** An overloaded or broken AI makes weaker moves (the + heuristic, or a random move) instead of stopping games. + +## Overview + +```mermaid +flowchart TB + subgraph docker["Layer 1: Docker caps (compose.yaml)"] + direction TB + subgraph server["wizard_server — 1 CPU / 512 MB"] + U["uvicorn: max message 64 KB,
max 1000 connections"] + A["Layer 2: abuse limits
connections, message rate,
lobby creation (limits.py)"] + L["Layer 3: load monitor +
admission control (load.py)"] + B["AI bridge: in-flight cap,
circuit breaker (ai_bridge.py)"] + end + subgraph ai["wizard_ai — 1 CPU / 1 GB"] + Q["Layer 4: workers, queue,
deadlines, degradation
(service/client.py)"] + end + W["wizard_website — 0.25 CPU / 64 MB"] + end + Browser -->|websocket| U --> A --> L + L -->|AI turns| B -->|decide| Q + L -. "server_status" .-> Browser +``` + +## Layer 1 — Docker caps + +Set in `compose.yaml` from `.env` (defaults shown): + +| Container | CPU | Memory | Processes | +|-----------|-----|--------|-----------| +| `wizard_server` | `SERVER_CPUS=1.0` | `SERVER_MEMORY=512M` | 256 | +| `wizard_ai` | `AI_CPUS=1.0` | `AI_MEMORY=1G` | 256 | +| `wizard_website` | `WEBSITE_CPUS=0.25` | `WEBSITE_MEMORY=64M` | 64 | + +- **CPU** is a hard quota: when the container uses it up, the kernel + throttles it. The server notices this (see layer 3) and refuses new games. +- **Memory** is a hard limit: a container exceeding it is killed by the kernel + and restarted by Docker (`restart: unless-stopped`). For the server this + would end all games (state is in memory), so the limit has lots of headroom: + the server uses ~80 MB idle and ~100 MB with 100 games (see + [measurements](#measurements)). +- **Processes** (`pids_limit`) stop fork bombs / thread explosions. +- The AI container sets `OMP_NUM_THREADS` / `MKL_NUM_THREADS` to `AI_WORKERS`, + so numeric libraries (numpy, torch) don't start one thread per *host* core + inside a 1-CPU container. + +uvicorn (`wizard_server/Dockerfile`) adds transport limits: +`--ws-max-size 65536` (larger messages close the connection with code 1009; +real messages are < 1 KB), `--ws-max-queue 16`, `--limit-concurrency 1000` +(HTTP 503 beyond that — a last resort above the app's own connection limit). + +## Layer 2 — abuse limits (server) + +`wizard_server/app/limits.py`, applied in `WizardService.handle_client` +(`service.py`): + +| Limit | Setting (default) | When exceeded | +|-------|-------------------|---------------| +| Open connections in total | `MAX_CONNECTIONS` (500) | `error` code `busy`, close **1013** "Try Again Later" | +| Open connections per client IP | `MAX_CONNECTIONS_PER_IP` (10) | same | +| Messages per connection | `MESSAGE_RATE` (10/s), `MESSAGE_BURST` (30) | message dropped; `error` code `rate_limited` (only on the 1st and every 10th drop) | +| Flooding | `MAX_RATE_VIOLATIONS` (50 drops) | connection closed with **1008** | +| Lobbies created per client IP | `LOBBY_CREATE_RATE` (10 per minute) | `error` code `rate_limited` | + +Building blocks: + +- `TokenBucket(rate, burst)` — the classic algorithm: a bucket holds up to + `burst` tokens, refills at `rate` per second, each action takes one token. + Allows short bursts (a player clicking quickly) but bounds the average. +- `KeyedRateLimiter` — one bucket per key (client IP); buckets that refilled + completely are forgotten by the cleanup task every 30 s. +- `ConnectionLimiter` — counts open connections in total and per IP; released + in the `finally` block of `handle_client`, so crashes cannot leak slots. + +**Client IP**: `websocket.client.host`. Behind Caddy, uvicorn's +`--proxy-headers` sets it from `X-Forwarded-For`; Caddy ignores +client-supplied `X-Forwarded-For` values, so the IP cannot be spoofed. In +local development everything comes from `127.0.0.1`. + +Normal play needs about one message per second, so these limits are never +noticeable for humans. Several players behind one NAT (e.g. a household) +share the per-IP connection limit — raise `MAX_CONNECTIONS_PER_IP` if that is +a realistic use case. + +## Layer 3 — load monitor and admission control (server) + +### Detecting overload (`load.py`) + +`LoadMonitor.run()` is started in the app lifespan and takes a sample every +`LOAD_SAMPLE_INTERVAL` (1 s): + +| Signal | How it is measured | Why | +|--------|--------------------|-----| +| **Event loop lag** | the monitor sleeps 1 s and measures how much later it woke up | the most direct measure of "the server reacts slowly"; works everywhere | +| **CPU** | process CPU time per second, relative to the container's CPU quota (at most 1 core, because game logic runs on one core) | detects saturation before players notice lag | +| **Throttling** | `nr_throttled / nr_periods` from `cpu.stat` | share of scheduling periods in which the container hit its Docker CPU cap | + +CPU quota and throttling come from the process's **own cgroup** (cgroup v2): +`/proc/self/cgroup` gives its path, e.g. `0::/` inside a container (→ +`/sys/fs/cgroup`) or `0::/user.slice/…scope` on a host. Without cgroup files +(e.g. macOS) the monitor still works with lag and CPU against one core. + +Each sample is classified (`classify`), and `next_level` turns the recent +classifications into the current level with hysteresis: + +| Level | Entered | Left | +|-------|---------|------| +| `normal` | — | — | +| `busy` | one sample with lag ≥ `LOAD_LAG_BUSY` (0.1 s) or CPU ≥ `LOAD_CPU_BUSY` (70 %) | after `LOAD_RECOVER_SAMPLES` (10) normal samples in a row | +| `overloaded` | `LOAD_OVERLOAD_SAMPLES` (3) samples in a row with lag ≥ `LOAD_LAG_OVERLOAD` (0.25 s), CPU ≥ `LOAD_CPU_OVERLOAD` (90 %) or throttled ≥ `LOAD_THROTTLE_OVERLOAD` (50 %) — **or one** sample with lag ≥ `LOAD_LAG_SEVERE` (1 s) | after 10 samples in a row that are not overloaded | + +Why hysteresis: without it a server at the edge would flip between states +every second, and every flip is broadcast to all players. Why the "severe" +rule: when the event loop is extremely late, samples themselves arrive late +(one per several seconds), so waiting for three would delay detection a lot. + +`busy` is informational only. `LOAD_FORCE_LEVEL` pins the level for testing. + +### Admission control (`service.py`) + +`WizardService.accepting_games` is true unless the level is `overloaded` or +`MAX_ACTIVE_GAMES` (50) games are running (`LobbyManager.active_games()`). + +| Action | While not accepting games | +|--------|---------------------------| +| `create_lobby` | refused — `error` code `busy` | +| `start_game` | refused — `error` code `busy` | +| `join_lobby` | allowed (cheap) | +| moves in running games, `return_to_lobby`, leaving | allowed (protects running games) | +| reconnects (`hello` with token) | allowed | + +Refusing new games sheds load by itself: finished games are not replaced, so +the load falls and the server recovers without intervention. + +### Telling the players + +Every connected client gets a `server_status` message right after `welcome` +and whenever the level or `accepting_games` changes: + +```json +{"type": "server_status", "load": "overloaded", "accepting_games": false} +``` + +Changes come from the load monitor (`on_change`) and from lobbies +(`on_status_change` when a game starts, finishes or is abandoned), and are +de-duplicated so an unchanged status is not re-sent. + +### Health endpoint + +`GET /health` (public: `https://game.marvinkrausser.com/wizard/api/health`): + +```json +{ + "status": "ok", + "load": "normal", "loop_lag": 0.0003, "cpu": 0.012, "throttled": 0.0, + "accepting_games": true, + "lobbies": 4, "active_games": 3, "sessions": 9, "connections": 7, + "ai_available": true, + "ai": {"connections": 1, "pending": 0, "paused": false} +} +``` + +`status` **stays "ok" under load** on purpose: the Docker health check must +never mark an overloaded server unhealthy, because a restart would end every +running game. + +## Layer 4 — AI cost bounds + +### In the server (`ai_bridge.py`) + +| Mechanism | Setting (default) | Behaviour | +|-----------|-------------------|-----------| +| Deadline | `AI_DECISION_TIMEOUT` (5 s) | sent as `timeout` in every request; no answer in time → random move | +| In-flight cap | `AI_MAX_INFLIGHT` (64) | if that many requests already wait on an AI connection, the move is played randomly without asking. Keep it ≥ `MAX_ACTIVE_GAMES`: every game has at most one AI deciding at a time | +| Circuit breaker | `AI_BREAKER_FAILURES` (5), `AI_BREAKER_COOLDOWN` (30 s) | after 5 **real failures** in a row (timeouts, lost connection, malformed/illegal answers, unexpected errors) the AI is not asked for 30 s; games play random AI moves instead of waiting 5 s per move | + +What counts as a failure matters: the breaker is for a *broken or hanging* AI +service. Deliberate load shedding — the in-flight cap, or the AI answering +`overloaded` / `expired` — is **not** a failure: the service is healthy and +answered immediately, and pausing it would only make play worse. + +### In the AI service (`wizard_ai/wizard_ai/service/client.py`) + +```mermaid +flowchart TD + R[decide request] --> Q{"in progress ≥
AI_MAX_QUEUE?"} + Q -- yes --> E1["error: overloaded"] + Q -- no --> W[wait for a free worker
AI_WORKERS] + W --> X{"server timeout
already passed?"} + X -- yes --> E2["error: expired"] + X -- no --> D{"≥ AI_DEGRADE_QUEUE others waiting,
or primary agent slower than
half the remaining time?"} + D -- yes --> H[heuristic agent] + D -- no --> M[primary agent / model] + H --> OK["action + agent name"] + M --> OK +``` + +| Setting (default) | Meaning | +|-------------------|---------| +| `AI_WORKERS` (1) | decisions computed at the same time; agents must be thread safe if > 1 | +| `AI_MAX_QUEUE` (64) | requests in progress (waiting + computing) before refusing with `overloaded` | +| `AI_DEGRADE_QUEUE` (8) | from this many other waiting requests, the heuristic decides instead of the model | + +The service also keeps a smoothed average of the primary agent's decision +time; if it exceeds half of the time left before the server's deadline, the +heuristic answers instead. The reply's `agent` field says who decided (the +server logs it at debug level). With the heuristic as primary agent (today's +default without a trained model) there is nothing cheaper to fall back to, so +degradation only matters once a real model is deployed. + +## What players see + +| Situation | Start page | Lobby | Running game | +|-----------|-----------|-------|--------------| +| Normal | — | — | — | +| Not accepting games (overloaded or game limit) | notice "The server is busy…", *Create lobby* disabled, *Join* works | notice, *Start game* shows "Server busy" (disabled) | unchanged | +| Connection limit reached | red banner "The server is full right now. Retrying in a moment…", retries after 15 s, 30 s, 60 s … | — | — | +| Message rate exceeded | toast "Too many messages, slow down." | same | same | +| AI overloaded / broken | — | — | AI players make weaker (heuristic or random) moves | + +Implemented in `useWizard.ts` (`serverStatus`, connection status `busy`), +`App.tsx`, `HomeScreen.tsx` and `LobbyScreen.tsx`. + +## Configuration reference + +All settings with their defaults. Server settings are read in +`wizard_server/app/config.py`, AI settings in `wizard_ai/wizard_ai/config.py`. +Settings marked *compose* are already wired through `compose.yaml` / `.env`; +the others can be added to the `environment:` section of the service. + +| Variable | Component | Default | Wired in compose | +|----------|-----------|---------|------------------| +| `SERVER_CPUS` / `SERVER_MEMORY` | Docker | 1.0 / 512M | yes | +| `AI_CPUS` / `AI_MEMORY` | Docker | 1.0 / 1G | yes | +| `WEBSITE_CPUS` / `WEBSITE_MEMORY` | Docker | 0.25 / 64M | yes | +| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | 500 / 10 | yes | +| `MESSAGE_RATE` / `MESSAGE_BURST` | server | 10 / 30 | no | +| `MAX_RATE_VIOLATIONS` | server | 50 | no | +| `LOBBY_CREATE_RATE` | server | 10 per minute | no | +| `MAX_LOBBIES` | server | 1000 | no | +| `MAX_ACTIVE_GAMES` | server | 50 | yes | +| `LOAD_SAMPLE_INTERVAL` | server | 1.0 s | no | +| `LOAD_LAG_BUSY` / `LOAD_LAG_OVERLOAD` / `LOAD_LAG_SEVERE` | server | 0.1 / 0.25 / 1.0 s | no | +| `LOAD_CPU_BUSY` / `LOAD_CPU_OVERLOAD` | server | 0.7 / 0.9 | no | +| `LOAD_THROTTLE_OVERLOAD` | server | 0.5 | no | +| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | server | 3 / 10 | no | +| `LOAD_FORCE_LEVEL` | server | empty | no | +| `AI_DECISION_TIMEOUT` | server | 5 s | no | +| `AI_MAX_INFLIGHT` | server | 64 | no | +| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | server | 5 / 30 s | no | +| `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | 1 / 64 / 8 | yes | + +## Measurements + +Measured on a development machine (uvicorn, heuristic AI, normal pacing: +`AI_MOVE_DELAY` 0.8 s, one scripted human per game taking 1.5 s per move, +4 players per game): + +| Running games | Server CPU | Server memory | AI service CPU | AI memory | +|---------------|-----------|---------------|----------------|-----------| +| 0 | ~0 % | ~55–80 MB | ~0 % | ~30 MB | +| 50 | 0.8 % of a core | ~88 MB | 0.5 % | ~30 MB | +| 100 | 1.5 % of a core | ~99 MB | 0.9 % | ~30 MB | + +Overload test: server limited to **15 % of one CPU** +(`systemd-run --user --scope -p CPUQuota=15%`), 30 games playing at full speed +(no delays, instant human moves): + +| Time | Event | +|------|-------| +| 0 s | load starts | +| ~1 s | `busy` | +| ~3 s | `overloaded` (CPU at 100 % of quota, throttled 100 %) → `create_lobby` refused with `busy` | +| … | running games keep playing (534 moves made while overloaded) | +| ~14 s | games that finished could not restart → load gone → back to `normal` | + +What this means: + +- With realistic pacing a game costs almost nothing; **1 CPU is far more than + 50 games need**. `MAX_ACTIVE_GAMES=50` is a conservative safety cap, not a + CPU requirement — raising it to a few hundred is fine with the heuristic AI. +- Overload only happens with abnormal traffic (scripts, floods) or a heavy AI + model. That is exactly what the limits are for. +- A neural-network model will be the most expensive part. Measure its + decision time and CPU with `python -m wizard_ai evaluate` and `docker stats` + before raising the caps. + +## Tuning guide + +| Symptom | Adjust | +|---------|--------| +| "Server busy" although `/health` shows low `cpu` and `load: normal` | the game limit is reached: raise `MAX_ACTIVE_GAMES` (and `AI_MAX_INFLIGHT` to at least the same value) | +| "Server busy" with high `cpu` / `throttled` | give the server more CPU (`SERVER_CPUS`) — or accept it: the server is protecting itself | +| Friends behind one router get "server full" | raise `MAX_CONNECTIONS_PER_IP` | +| Busy/overloaded flips too eagerly on a slow machine | raise `LOAD_LAG_*` thresholds or `LOAD_OVERLOAD_SAMPLES` | +| AI plays weaker under load (logs: "Busy … using heuristic agent") | raise `AI_CPUS` and `AI_WORKERS` (thread-safe agent!) or use a smaller model; `AI_DEGRADE_QUEUE` sets how early it degrades | +| Server logs "AI failed 5 times in a row, pausing AI requests" | the AI service hangs or crashes: check `docker compose logs wizard_ai` | +| Server logs "… AI requests already waiting" | more simultaneous AI turns than `AI_MAX_INFLIGHT`: raise it to ≥ `MAX_ACTIVE_GAMES` | +| Container restarts, `docker inspect` shows `OOMKilled: true` | raise `SERVER_MEMORY` / `AI_MEMORY` | + +## Monitoring + +- `curl -s https://game.marvinkrausser.com/wizard/api/health` — load, CPU, + games, connections, AI state (see above). +- `docker stats` — live CPU and memory per container against the caps. +- Log lines to watch (`docker compose logs -f wizard_server wizard_ai`): + + | Log line | Meaning | + |----------|---------| + | `Load level normal -> busy` / `busy -> overloaded` / `-> normal` | load level changes (with lag, CPU, throttling) | + | `Refusing connection from : connection limit reached` | connection limits hit | + | `Closing connection from : message flood` | a client was cut off | + | `AI failed N times in a row, pausing AI requests for 30s` | circuit breaker opened | + | `Playing random move for player N: …` | an AI move fell back to random (reason included) | + | AI: `Overloaded: N requests in progress` / `Dropping expired request` / `Busy (N waiting): using heuristic agent` | the AI service shedding load | + +## Testing it yourself + +Automated tests: + +- `wizard_server/tests/test_limits.py` — token bucket, limiters, load levels + and hysteresis, cgroup parsing, AI circuit breaker and in-flight cap. +- `wizard_server/tests/test_protection.py` — floods, connection limits, lobby + creation rate, forced overload, `MAX_ACTIVE_GAMES`, a game continuing while + overloaded, `server_status` messages. +- `wizard_ai/tests/test_ai.py` — queue full, expired requests, degradation. + +Manual checks: + +```bash +# Refusal UI without real load +cd wizard_server && LOAD_FORCE_LEVEL=overloaded AI_SHARED_SECRET=dev uvicorn app.main:app + +# Game limit: start one game, then a second lobby's "Start game" is refused +cd wizard_server && MAX_ACTIVE_GAMES=1 AI_SHARED_SECRET=dev uvicorn app.main:app + +# Real CPU exhaustion on Linux: cap the server at 15 % of a CPU, +# then run many games without delays (e.g. a small websocket script) +cd wizard_server && AI_MOVE_DELAY=0 TRICK_PAUSE=0 MAX_CONNECTIONS_PER_IP=200 \ + MESSAGE_RATE=1000 MESSAGE_BURST=1000 AI_SHARED_SECRET=dev \ + systemd-run --user --scope -p CPUQuota=15% uvicorn app.main:app +``` + +## Known limitations + +- A running game whose humans all disconnected keeps its `MAX_ACTIVE_GAMES` + slot until their sessions expire (`SESSION_TTL`, 30 min), because they may + reconnect. It costs no CPU meanwhile. +- The limits are per server process. That is fine because the server must run + as a single process anyway (all state is in memory). +- Players behind one IP share `MAX_CONNECTIONS_PER_IP` and `LOBBY_CREATE_RATE`. +- There is no waiting queue: players have to retry when the server is busy. diff --git a/docs/README.md b/docs/README.md index ccf475b..5165400 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,11 +11,12 @@ to be read top to bottom the first time and used as a reference afterwards. | 1 | [Architecture overview](01-architecture.md) | The four components, how a move travels through the system, repository layout, key design decisions | | 2 | [Game rules as implemented](02-game-rules.md) | Every rule of Wizard as the code implements it, with the exact function that enforces it | | 3 | [Game engine (`wizard_game`)](03-game-engine.md) | The rules package: cards, the `WizardGame` state machine, observations, actions, events | -| 4 | [Game server (`wizard_server/app`)](04-server.md) | Sessions, reconnects, lobbies, running games, AI turns, broadcasting, configuration | +| 4 | [Game server (`wizard_server/app`)](04-server.md) | Sessions, reconnects, lobbies, running games, AI turns, broadcasting, limits (summary), configuration | | 5 | [Protocol reference](05-protocol.md) | Every websocket message between browser ↔ server and server ↔ AI, with examples | | 6 | [Frontend (`wizard_website`)](06-frontend.md) | The React app: connection hook, screens, components, styling, build | | 7 | [AI player (`wizard_ai`)](07-ai.md) | Agents, feature encoding, policy interface, training, evaluation, production service — and how to plug in a real model | | 8 | [Deployment, development & operations](08-operations.md) | Docker, networks, Caddy, environment variables, local setup, tests, troubleshooting, how-to recipes | +| 9 | [Limits, load control and performance](09-limits-and-performance.md) | Resource caps, abuse limits, overload detection, what happens when compute runs out, AI cost bounds, measurements, tuning and monitoring | ## The system in one paragraph diff --git a/wizard_ai/README.md b/wizard_ai/README.md index c31d619..b3306ba 100644 --- a/wizard_ai/README.md +++ b/wizard_ai/README.md @@ -16,12 +16,16 @@ provides everything around it. `Policy.update`, with checkpointing - `training/evaluate.py` — play agents against each other (mean score, win rate) - `service/client.py` — production: connects to `ws://wizard_server:8083/internal/ai`, - answers `decide` requests, reconnects automatically + answers `decide` requests, reconnects automatically. Work is bounded: `AI_WORKERS` (1) + parallel decisions, at most `AI_MAX_QUEUE` (64) requests in progress (more are answered + "overloaded"), expired requests are dropped, and from `AI_DEGRADE_QUEUE` (8) waiting requests — + or when the model is too slow — the heuristic decides instead of the model + (see `../docs/09-limits-and-performance.md`) ```bash pip install -r requirements-dev.txt python -m wizard_ai train --episodes 5000 --out models/latest python -m wizard_ai evaluate --agents model,heuristic,random,random --games 200 -python -m wizard_ai serve # env: SERVER_URL, AI_SHARED_SECRET, AGENT, MODEL_PATH +python -m wizard_ai serve # env: SERVER_URL, AI_SHARED_SECRET, AGENT, MODEL_PATH, AI_WORKERS, AI_MAX_QUEUE, AI_DEGRADE_QUEUE pytest ``` diff --git a/wizard_server/README.md b/wizard_server/README.md index 3225567..9b5ad23 100644 --- a/wizard_server/README.md +++ b/wizard_server/README.md @@ -8,11 +8,19 @@ FastAPI game server. It manages sessions, lobbies and games, and talks to the AI - `app/protocol.py` — client message schema - `app/service.py` — dispatches client messages, reconnects sessions, cleanup - `app/lobby.py` — lobbies, seats, host actions, running games and AI turns -- `app/ai_bridge.py` — request/response protocol with the AI service, random-move fallback +- `app/ai_bridge.py` — request/response protocol with the AI service, random-move fallback, + in-flight cap and circuit breaker - `app/sessions.py` — sessions (token based, survive reloads) and ordered websocket sending +- `app/limits.py` — token bucket, per-IP rate limiter, connection limiter (abuse protection) +- `app/load.py` — load monitor (event loop lag, CPU, cgroup throttling) with hysteresis; the + service refuses new lobbies/games while overloaded or at `MAX_ACTIVE_GAMES` -Configuration (environment): `AI_SHARED_SECRET`, `AI_DECISION_TIMEOUT` (5s), `AI_MOVE_DELAY` (0.8s), -`TRICK_PAUSE` (1.8s), `SESSION_TTL` (1800s), `MAX_LOBBIES` (1000), `ALLOWED_ORIGINS`, `LOG_LEVEL`. +Configuration (environment, all in `app/config.py`): `AI_SHARED_SECRET`, `AI_DECISION_TIMEOUT` (5s), +`AI_MOVE_DELAY` (0.8s), `TRICK_PAUSE` (1.8s), `SESSION_TTL` (1800s), `MAX_LOBBIES` (1000), +`ALLOWED_ORIGINS`, `LOG_LEVEL`, and the limits: `MAX_CONNECTIONS` (500), `MAX_CONNECTIONS_PER_IP` (10), +`MESSAGE_RATE`/`MESSAGE_BURST` (10/30), `LOBBY_CREATE_RATE` (10/min), `MAX_ACTIVE_GAMES` (50), +`LOAD_*` thresholds, `AI_MAX_INFLIGHT` (64), `AI_BREAKER_FAILURES`/`AI_BREAKER_COOLDOWN` (5/30s). +See `../docs/04-server.md` and `../docs/09-limits-and-performance.md`. ```bash pip install -r requirements-dev.txt diff --git a/wizard_website/README.md b/wizard_website/README.md index 3028fca..ecd91b1 100644 --- a/wizard_website/README.md +++ b/wizard_website/README.md @@ -18,9 +18,12 @@ development, or `VITE_WS_URL` at build time to use an absolute websocket URL ins ## Structure - `src/api/` — protocol types and the `useWizard` hook (websocket connection, - reconnect with a per-tab session token, game state) + reconnect with a per-tab session token, game state, server load status; slow + retries when the server is full) - `src/components/` — reusable UI (button, playing card, modal, toasts) -- `src/screens/HomeScreen` — enter a name, create or join a lobby -- `src/screens/LobbyScreen` — lobby code, players, AI players, rule options, start +- `src/screens/HomeScreen` — enter a name, create or join a lobby (Create is disabled with a + notice while the server is busy) +- `src/screens/LobbyScreen` — lobby code, players, AI players, rule options, start (disabled + with a notice while the server is busy) - `src/screens/GameScreen` — the table: seats, trick, hand, bidding, trump choice, scoreboard, round summary, game over