marvin/Wizard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document limits, load control and performance

Browse Source 
- New docs/09-limits-and-performance.md: principles, all protection
  layers, what players see, configuration reference, measurements,
  tuning guide, monitoring and how to test
- Server chapter summarises the limits and links to chapter 9; circuit
  breaker description and AI_MAX_INFLIGHT default corrected
- Architecture: new files in the layout, design decisions for load
  handling; docs index lists chapter 9
- Protocol, AI and operations chapters link to chapter 9; test counts
- Root and component READMEs describe the limits and settings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
marvin
2026-09-27 13:04:21 +02:00
co-authored by Claude Opus 5.5
parent 763c5678f1
commit 9ffee8b952
11 changed files with 462 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+16 -1
View File
@@ -33,10 +33,25 @@ The `Observation` / action JSON format is defined once in `wizard_game` and shar
frontend, the server and the AI, so AI models see exactly the same data in training and in frontend, the server and the AI, so AI models see exactly the same data in training and in
production. production.
## Limits and load control
Wizard shares its host with other services, so its resource use is bounded in layers
(details: [`docs/09-limits-and-performance.md`](docs/09-limits-and-performance.md)):
- **Docker caps** per container (server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB),
set in `.env`.
- **Abuse limits**: connections per IP and in total, messages per second per connection,
lobbies per IP per minute.
- **Overload handling**: the server watches event loop lag, CPU and cgroup throttling. While it
is overloaded, or `MAX_ACTIVE_GAMES` games are running, new lobbies and game starts are refused
("server busy" in the UI). Running games always continue.
- **AI cost bounds**: bounded queue and workers in the AI service, which falls back to the cheap
heuristic player when busy; circuit breaker and in-flight cap in the server.
## Deployment ## Deployment
```bash ```bash
cp .env.example .env # set AI_SHARED_SECRET cp .env.example .env # set AI_SHARED_SECRET (limits optional)
docker network create wizard_website # if they don't exist yet docker network create wizard_website # if they don't exist yet
docker network create wizard_api docker network create wizard_api
docker compose up -d --build docker compose up -d --build
docs/01-architecture.md
+12 -6
View File
@@ -5,7 +5,7 @@
| Component | Folder | Language / stack | Runs as | Responsibility | | Component | Folder | Language / stack | Runs as | Responsibility |
|-----------|--------|------------------|---------|----------------| |-----------|--------|------------------|---------|----------------|
| Website | `wizard_website/` | React 19, TypeScript, Vite, CSS modules | static files served by nginx (container port 8082) | UI: start page, lobby, game table | | Website | `wizard_website/` | React 19, TypeScript, Vite, CSS modules | static files served by nginx (container port 8082) | UI: start page, lobby, game table |
| Game server | `wizard_server/app/` | Python 3.13, FastAPI, uvicorn | one process (container port 8083) | sessions, lobbies, running games, asking the AI for moves | | Game server | `wizard_server/app/` | Python 3.13, FastAPI, uvicorn | one process (container port 8083) | sessions, lobbies, running games, asking the AI for moves, limits and load control |
| Game engine | `wizard_server/game_logic/` (package `wizard_game`) | pure Python, no dependencies | library, installed into server and AI images | the rules of Wizard | | Game engine | `wizard_server/game_logic/` (package `wizard_game`) | pure Python, no dependencies | library, installed into server and AI images | the rules of Wizard |
| AI service | `wizard_ai/` | Python, `websockets` | one process, no listening port | chooses moves for AI seats; training tools | | AI service | `wizard_ai/` | Python, `websockets` | one process, no listening port | chooses moves for AI seats; training tools |
@@ -89,8 +89,8 @@ Important properties of this flow:
``` ```
. .
├── compose.yaml # all three containers + networks ├── compose.yaml # all three containers, networks, CPU/memory caps
├── .env.example # template for .env (AI_SHARED_SECRET, …) ├── .env.example # template for .env (secret, resource limits, capacity)
├── README.md # short project readme ├── README.md # short project readme
├── docs/ # this documentation ├── docs/ # this documentation
├── instructions.txt # original task description ├── instructions.txt # original task description
@@ -107,8 +107,11 @@ Important properties of this flow:
│ │ ├── service.py # WizardService: dispatches messages, sessions, cleanup │ │ ├── service.py # WizardService: dispatches messages, sessions, cleanup
│ │ ├── sessions.py # Session, Connection (ordered send queue), SessionRegistry │ │ ├── sessions.py # Session, Connection (ordered send queue), SessionRegistry
│ │ ├── lobby.py # Lobby, Seat, LobbyManager, AI turn loop │ │ ├── lobby.py # Lobby, Seat, LobbyManager, AI turn loop
│ │ └── ai_bridge.py # request/response with the AI service + fallback │ │ ├── ai_bridge.py # request/response with the AI service, fallback, circuit breaker
│ ├── tests/test_server.py # websocket-level tests of the server │ │ ├── limits.py # TokenBucket, rate and connection limiters
│ │ └── load.py # LoadMonitor: detects overload (lag, CPU, cgroup throttling)
│ ├── tests/ # conftest.py/helpers.py (fixtures), test_server.py,
│ │ # test_protection.py (limits/overload), test_limits.py (units)
│ └── game_logic/ # installable package "wizard-game" │ └── game_logic/ # installable package "wizard-game"
│ ├── pyproject.toml │ ├── pyproject.toml
│ ├── wizard_game/ │ ├── wizard_game/
@@ -130,7 +133,7 @@ Important properties of this flow:
│ │ ├── agents/ # Agent interface, random, heuristic, model agent │ │ ├── agents/ # Agent interface, random, heuristic, model agent
│ │ ├── model/ # encoding.py (features/actions), policy.py (model interface) │ │ ├── model/ # encoding.py (features/actions), policy.py (model interface)
│ │ ├── training/ # env.py, self_play.py, evaluate.py │ │ ├── training/ # env.py, self_play.py, evaluate.py
│ │ └── service/client.py # production websocket client │ │ └── service/client.py # production websocket client (workers, queue, degradation)
│ └── tests/test_ai.py │ └── tests/test_ai.py
│ │
└── wizard_website/ └── wizard_website/
@@ -158,3 +161,6 @@ Important properties of this flow:
| Session token in `sessionStorage` | Survives page reloads (reconnect to the same seat) but is per tab, so you can test several players in several tabs. | | Session token in `sessionStorage` | Survives page reloads (reconnect to the same seat) but is per tab, so you can test several players in several tabs. |
| Per-connection send queue | Websocket messages are always sent in order and never concurrently (see `sessions.Connection`). | | Per-connection send queue | Websocket messages are always sent in order and never concurrently (see `sessions.Connection`). |
| Relative asset paths + relative API URL in the frontend | The app works under any path prefix (`/wizard/website/` in production, `/` in development) without rebuilding. | | Relative asset paths + relative API URL in the frontend | The app works under any path prefix (`/wizard/website/` in production, `/` in development) without rebuilding. |
| Refuse new games under load, never interrupt running ones | Players in a game are never punished for load; creating/starting games is what adds load, so that is what gets refused ("server busy"). Refusing sheds load by itself. See [chapter 9](09-limits-and-performance.md). |
| Hard Docker caps plus software limits | The caps guarantee the host is protected even if a software limit fails; the software limits make overload visible and graceful instead of just slow. |
| Health check stays "ok" under load | Docker must never restart an overloaded server — that would end every running game. |
docs/04-server.md
+26 -88
View File
@@ -256,97 +256,35 @@ The AI service keeps a websocket open to `/internal/ai`. `AIBridge`:
server has already given up on; server has already given up on;
- **in-flight cap**: if `AI_MAX_INFLIGHT` requests are already waiting on a - **in-flight cap**: if `AI_MAX_INFLIGHT` requests are already waiting on a
connection, the move is played by the fallback without asking; connection, the move is played by the fallback without asking;
- **circuit breaker**: after `AI_BREAKER_FAILURES` failures in a row (timeouts, - **circuit breaker**: after `AI_BREAKER_FAILURES` real failures in a row
errors, illegal answers) the AI is not asked for `AI_BREAKER_COOLDOWN` (timeouts, lost connection, malformed/illegal answers, unexpected errors) the
seconds, so a struggling AI service is not buried under more requests. AI is not asked for `AI_BREAKER_COOLDOWN` seconds, so games don't wait
`AI_DECISION_TIMEOUT` for every move of a hanging AI. Load shedding (the
in-flight cap, AI answers `overloaded` / `expired`) does not count.
## Limits and overload handling ## Limits and overload handling
The server protects the host and its players in three layers. The guiding rule The complete description — all layers, thresholds, what players see, tuning,
is: **running games always continue; new work is refused when there is no monitoring and measurements — is in
capacity; abuse is rejected immediately.** [chapter 9: Limits, load control and performance](09-limits-and-performance.md).
In short, for the server:
```mermaid - **Rule**: running games always continue; new lobbies/games are refused when
flowchart TD there is no capacity; abuse is rejected immediately.
C[new websocket] --> L1{"connection limit<br/>total / per IP"} - **Abuse limits** (`limits.py`, applied in `WizardService.handle_client`):
L1 -- over --> R1["error busy + close 1013"] connections in total and per IP (refused with `error` code `busy` + close
L1 -- ok --> M[message] 1013), a token bucket per connection for messages (`rate_limited`, close
M --> L2{"message rate<br/>token bucket"} 1008 on floods), lobby creation per IP.
L2 -- over --> R2["dropped + error rate_limited<br/>(close 1008 when flooding)"] - **Load monitor** (`load.py`, `LoadMonitor.run()` in the app lifespan):
L2 -- ok --> D{"create_lobby /<br/>start_game?"} samples event loop lag, CPU against the container quota, and cgroup
D -- no --> OK[handled normally] throttling every second; levels `normal` / `busy` / `overloaded` with
D -- yes --> L3{"accepting_games?<br/>not overloaded and<br/>running games < MAX_ACTIVE_GAMES"} hysteresis.
L3 -- no --> R3["error busy"] - **Admission control** (`WizardService.accepting_games`): `create_lobby` and
L3 -- yes --> OK `start_game` are refused while overloaded or while `MAX_ACTIVE_GAMES` games
``` run; everything else keeps working. Changes are broadcast to all clients as
`server_status`.
### 1. Hard limits (Docker and uvicorn) - `/health` reports the load but its `status` stays "ok", so Docker never
restarts an overloaded server (which would end all games).
- `compose.yaml` caps every container (`deploy.resources.limits` + `pids_limit`):
server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB by default,
overridable in `.env` (`SERVER_CPUS`, `SERVER_MEMORY`, …).
- uvicorn (`wizard_server/Dockerfile`): `--ws-max-size 65536` (bigger messages
close the connection with 1009), `--ws-max-queue 16`,
`--limit-concurrency 1000` (HTTP 503 beyond that — a last resort above the
app's own connection limit).
### 2. Abuse limits (`limits.py`, applied in `WizardService`)
| Limit | Default | When exceeded |
|-------|---------|---------------|
| open connections total / per client IP | 500 / 10 | `error` (code `busy`), then close **1013** "Try Again Later" |
| messages per connection (token bucket) | 10/s, burst 30 | message dropped, `error` (code `rate_limited`, not repeated for every message); after 50 drops the connection is closed with 1008 |
| lobbies created per client IP | 10 per minute | `error` (code `rate_limited`) |
The client IP is `websocket.client.host`. Behind Caddy, uvicorn's
`--proxy-headers` sets it from `X-Forwarded-For` (Caddy does not pass on
client-supplied values, so it cannot be spoofed).
### 3. Load monitor and admission control (`load.py`)
`LoadMonitor.run()` (started in the app lifespan) samples once per second:
- **event loop lag** — how late the loop wakes up; the most direct "players
notice slowness" signal;
- **CPU** — process CPU time relative to the container's CPU quota (read from
the process's own cgroup, `/proc/self/cgroup` → `…/cpu.max`), at most one
core because game logic runs on one core;
- **throttling** — share of CPU periods in which the container hit its limit
(`cpu.stat`: `nr_throttled / nr_periods`).
Each sample is classified (`classify`) and `next_level` derives the level with
hysteresis:
| Level | Entered when | Left when |
|-------|--------------|-----------|
| `busy` | one sample with lag ≥ 0.1 s or CPU ≥ 70 % | 10 normal samples in a row |
| `overloaded` | 3 samples in a row with lag ≥ 0.25 s, CPU ≥ 90 % or throttled ≥ 50 % — or **one** sample with lag ≥ 1 s | 10 samples in a row that are not overloaded |
`busy` is only informational. While **overloaded**, or while
`MAX_ACTIVE_GAMES` (default 50) games are running, `accepting_games` is false:
| Action | Refused when not accepting games |
|--------|----------------------------------|
| `create_lobby` | yes (`error` code `busy`) |
| `start_game` | yes (`error` code `busy`) |
| `join_lobby`, moves, `return_to_lobby`, reconnects | never — they are cheap and keep running games alive |
Refusing new games sheds load by itself: finished games are not replaced, so
the load drops and the server recovers.
Every connected client receives a `server_status` message after `welcome` and
whenever the level or `accepting_games` changes (the website then disables
"Create lobby" / "Start game" and shows a notice). `/health` reports `load`,
`loop_lag`, `cpu`, `throttled`, `accepting_games`, `active_games`,
`connections` and AI bridge stats, but its `status` **stays "ok"** under load:
an overloaded server must never be restarted by Docker, which would end all
running games.
Measured on a server capped at 15 % of a CPU with 30 games played at full
speed: overloaded after ~3 s, new lobbies refused, running games kept going,
back to normal ~10 s after the load dropped.
For manual testing, `LOAD_FORCE_LEVEL=overloaded` pins the level.
## Configuration (`config.py`) ## Configuration (`config.py`)
@@ -373,7 +311,7 @@ All values are read from environment variables when the module is imported.
| `LOAD_THROTTLE_OVERLOAD` | 0.5 | throttled share of CPU periods | | `LOAD_THROTTLE_OVERLOAD` | 0.5 | throttled share of CPU periods |
| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | 3 / 10 | hysteresis | | `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | 3 / 10 | hysteresis |
| `LOAD_FORCE_LEVEL` | empty | pin the level (testing) | | `LOAD_FORCE_LEVEL` | empty | pin the level (testing) |
| `AI_MAX_INFLIGHT` | 32 | waiting AI requests per AI connection | | `AI_MAX_INFLIGHT` | 64 | waiting AI requests per AI connection (keep ≥ `MAX_ACTIVE_GAMES`) |
| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | 5 / 30 s | AI circuit breaker | | `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | 5 / 30 s | AI circuit breaker |
## Housekeeping ## Housekeeping
docs/05-protocol.md
+4 -3
View File
@@ -192,8 +192,9 @@ Rules:
- No answer within `AI_DECISION_TIMEOUT` (5 s) → random move. The AI service - No answer within `AI_DECISION_TIMEOUT` (5 s) → random move. The AI service
drops requests whose `timeout` passed before it could start them (`expired`). drops requests whose `timeout` passed before it could start them (`expired`).
- The server stops asking an AI connection that already has `AI_MAX_INFLIGHT` - The server stops asking an AI connection that already has `AI_MAX_INFLIGHT`
unanswered requests, and pauses the AI after repeated failures (circuit unanswered requests, and pauses the AI after repeated real failures (circuit
breaker); see "Limits and overload handling" in breaker). `overloaded` / `expired` answers are load shedding, not failures,
[the server chapter](04-server.md). and do not trip the breaker. See
"Layer 4" in [chapter 9](09-limits-and-performance.md).
- Several AI service instances may connect; the server distributes requests - Several AI service instances may connect; the server distributes requests
round robin. round robin.
docs/07-ai.md
+3 -1
View File
@@ -268,7 +268,9 @@ count as a win for everyone tied). Seeded for fair comparisons.
primary agent already is the heuristic. The reply's `agent` field says which primary agent already is the heuristic. The reply's `agent` field says which
one decided; one decided;
- `OMP_NUM_THREADS` / `MKL_NUM_THREADS` are set to `AI_WORKERS` in - `OMP_NUM_THREADS` / `MKL_NUM_THREADS` are set to `AI_WORKERS` in
`compose.yaml`, so numeric libraries don't start a thread per host core; `compose.yaml`, so numeric libraries don't start a thread per host core
(the whole picture, including the server side, is in
[chapter 9](09-limits-and-performance.md));
- errors while deciding are reported back as `error` (the server then plays a - errors while deciding are reported back as `error` (the server then plays a
random move) instead of crashing; random move) instead of crashing;
- reconnects forever with exponential backoff (1 s … 30 s); logs a hint if the - reconnects forever with exponential backoff (1 s … 30 s); logs a hint if the
docs/08-operations.md
+3 -3
View File
@@ -51,8 +51,8 @@ not read by anything.
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | no (500 / 10) | open websocket connections | | `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | no (500 / 10) | open websocket connections |
| `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | no (1 / 64 / 8) | parallel decisions, waiting requests before refusing, queue depth from which the heuristic is used | | `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | no (1 / 64 / 8) | parallel decisions, waiting requests before refusing, queue depth from which the heuristic is used |
How the limits work together is described in "Limits and overload handling" How the limits work together, the full list of settings, measurements and a
in [the server chapter](04-server.md). Further server settings tuning guide are in [chapter 9](09-limits-and-performance.md). Further server settings
(`AI_MOVE_DELAY`, `TRICK_PAUSE`, load thresholds, …, see (`AI_MOVE_DELAY`, `TRICK_PAUSE`, load thresholds, …, see
"Configuration" in [the server chapter](04-server.md)) can be added to "Configuration" in [the server chapter](04-server.md)) can be added to
the `environment:` section of `wizard_server` in `compose.yaml`. the `environment:` section of `wizard_server` in `compose.yaml`.
@@ -155,7 +155,7 @@ cd wizard_website && npm run lint && npm run build
| Suite | Count | Covers | | Suite | Count | Covers |
|-------|-------|--------| |-------|-------|--------|
| game engine | 52 | rules, full random games for 3–6 players, invariants, JSON | | game engine | 52 | rules, full random games for 3–6 players, invariants, JSON |
| server | 32 | websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels | | server | 33 | websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels |
| AI | 17 | encoding, agents, env, trainer, checkpoints, service limits | | AI | 17 | encoding, agents, env, trainer, checkpoints, service limits |
## Troubleshooting ## Troubleshooting
docs/09-limits-and-performance.md
+373
View File
@@ -0,0 +1,373 @@
# 9. Limits, load control and performance
This chapter explains how Wizard keeps its resource use bounded, how it
protects itself against abuse, and what happens when it runs out of compute.
It covers all components, because the protection works in layers from Docker
down to the browser.
## Principles
1. **Running games always continue.** A game in progress is never stopped,
slowed down on purpose or refused a move because of load.
2. **New work is refused when there is no capacity.** Creating lobbies and
starting games is what adds load, so that is what gets refused — with a
clear "server busy" message, not a timeout. (A waiting queue was considered
and rejected as not worth the complexity.)
3. **Abuse is rejected immediately.** Floods and connection storms are cut off
at the door, never queued.
4. **The host is protected no matter what.** Hard Docker caps guarantee that
Wizard can never take CPU or memory from the other services on the machine,
even if every software limit fails.
5. **Degrade, don't fail.** An overloaded or broken AI makes weaker moves (the
heuristic, or a random move) instead of stopping games.
## Overview
```mermaid
flowchart TB
subgraph docker["Layer 1: Docker caps (compose.yaml)"]
direction TB
subgraph server["wizard_server — 1 CPU / 512 MB"]
U["uvicorn: max message 64 KB,<br/>max 1000 connections"]
A["Layer 2: abuse limits<br/>connections, message rate,<br/>lobby creation (limits.py)"]
L["Layer 3: load monitor +<br/>admission control (load.py)"]
B["AI bridge: in-flight cap,<br/>circuit breaker (ai_bridge.py)"]
end
subgraph ai["wizard_ai — 1 CPU / 1 GB"]
Q["Layer 4: workers, queue,<br/>deadlines, degradation<br/>(service/client.py)"]
end
W["wizard_website — 0.25 CPU / 64 MB"]
end
Browser -->|websocket| U --> A --> L
L -->|AI turns| B -->|decide| Q
L -. "server_status" .-> Browser
```
## Layer 1 — Docker caps
Set in `compose.yaml` from `.env` (defaults shown):
| Container | CPU | Memory | Processes |
|-----------|-----|--------|-----------|
| `wizard_server` | `SERVER_CPUS=1.0` | `SERVER_MEMORY=512M` | 256 |
| `wizard_ai` | `AI_CPUS=1.0` | `AI_MEMORY=1G` | 256 |
| `wizard_website` | `WEBSITE_CPUS=0.25` | `WEBSITE_MEMORY=64M` | 64 |
- **CPU** is a hard quota: when the container uses it up, the kernel
throttles it. The server notices this (see layer 3) and refuses new games.
- **Memory** is a hard limit: a container exceeding it is killed by the kernel
and restarted by Docker (`restart: unless-stopped`). For the server this
would end all games (state is in memory), so the limit has lots of headroom:
the server uses ~80 MB idle and ~100 MB with 100 games (see
[measurements](#measurements)).
- **Processes** (`pids_limit`) stop fork bombs / thread explosions.
- The AI container sets `OMP_NUM_THREADS` / `MKL_NUM_THREADS` to `AI_WORKERS`,
so numeric libraries (numpy, torch) don't start one thread per *host* core
inside a 1-CPU container.
uvicorn (`wizard_server/Dockerfile`) adds transport limits:
`--ws-max-size 65536` (larger messages close the connection with code 1009;
real messages are < 1 KB), `--ws-max-queue 16`, `--limit-concurrency 1000`
(HTTP 503 beyond that — a last resort above the app's own connection limit).
## Layer 2 — abuse limits (server)
`wizard_server/app/limits.py`, applied in `WizardService.handle_client`
(`service.py`):
| Limit | Setting (default) | When exceeded |
|-------|-------------------|---------------|
| Open connections in total | `MAX_CONNECTIONS` (500) | `error` code `busy`, close **1013** "Try Again Later" |
| Open connections per client IP | `MAX_CONNECTIONS_PER_IP` (10) | same |
| Messages per connection | `MESSAGE_RATE` (10/s), `MESSAGE_BURST` (30) | message dropped; `error` code `rate_limited` (only on the 1st and every 10th drop) |
| Flooding | `MAX_RATE_VIOLATIONS` (50 drops) | connection closed with **1008** |
| Lobbies created per client IP | `LOBBY_CREATE_RATE` (10 per minute) | `error` code `rate_limited` |
Building blocks:
- `TokenBucket(rate, burst)` — the classic algorithm: a bucket holds up to
`burst` tokens, refills at `rate` per second, each action takes one token.
Allows short bursts (a player clicking quickly) but bounds the average.
- `KeyedRateLimiter` — one bucket per key (client IP); buckets that refilled
completely are forgotten by the cleanup task every 30 s.
- `ConnectionLimiter` — counts open connections in total and per IP; released
in the `finally` block of `handle_client`, so crashes cannot leak slots.
**Client IP**: `websocket.client.host`. Behind Caddy, uvicorn's
`--proxy-headers` sets it from `X-Forwarded-For`; Caddy ignores
client-supplied `X-Forwarded-For` values, so the IP cannot be spoofed. In
local development everything comes from `127.0.0.1`.
Normal play needs about one message per second, so these limits are never
noticeable for humans. Several players behind one NAT (e.g. a household)
share the per-IP connection limit — raise `MAX_CONNECTIONS_PER_IP` if that is
a realistic use case.
## Layer 3 — load monitor and admission control (server)
### Detecting overload (`load.py`)
`LoadMonitor.run()` is started in the app lifespan and takes a sample every
`LOAD_SAMPLE_INTERVAL` (1 s):
| Signal | How it is measured | Why |
|--------|--------------------|-----|
| **Event loop lag** | the monitor sleeps 1 s and measures how much later it woke up | the most direct measure of "the server reacts slowly"; works everywhere |
| **CPU** | process CPU time per second, relative to the container's CPU quota (at most 1 core, because game logic runs on one core) | detects saturation before players notice lag |
| **Throttling** | `nr_throttled / nr_periods` from `cpu.stat` | share of scheduling periods in which the container hit its Docker CPU cap |
CPU quota and throttling come from the process's **own cgroup** (cgroup v2):
`/proc/self/cgroup` gives its path, e.g. `0::/` inside a container (→
`/sys/fs/cgroup`) or `0::/user.slice/…scope` on a host. Without cgroup files
(e.g. macOS) the monitor still works with lag and CPU against one core.
Each sample is classified (`classify`), and `next_level` turns the recent
classifications into the current level with hysteresis:
| Level | Entered | Left |
|-------|---------|------|
| `normal` | — | — |
| `busy` | one sample with lag ≥ `LOAD_LAG_BUSY` (0.1 s) or CPU ≥ `LOAD_CPU_BUSY` (70 %) | after `LOAD_RECOVER_SAMPLES` (10) normal samples in a row |
| `overloaded` | `LOAD_OVERLOAD_SAMPLES` (3) samples in a row with lag ≥ `LOAD_LAG_OVERLOAD` (0.25 s), CPU ≥ `LOAD_CPU_OVERLOAD` (90 %) or throttled ≥ `LOAD_THROTTLE_OVERLOAD` (50 %) — **or one** sample with lag ≥ `LOAD_LAG_SEVERE` (1 s) | after 10 samples in a row that are not overloaded |
Why hysteresis: without it a server at the edge would flip between states
every second, and every flip is broadcast to all players. Why the "severe"
rule: when the event loop is extremely late, samples themselves arrive late
(one per several seconds), so waiting for three would delay detection a lot.
`busy` is informational only. `LOAD_FORCE_LEVEL` pins the level for testing.
### Admission control (`service.py`)
`WizardService.accepting_games` is true unless the level is `overloaded` or
`MAX_ACTIVE_GAMES` (50) games are running (`LobbyManager.active_games()`).
| Action | While not accepting games |
|--------|---------------------------|
| `create_lobby` | refused — `error` code `busy` |
| `start_game` | refused — `error` code `busy` |
| `join_lobby` | allowed (cheap) |
| moves in running games, `return_to_lobby`, leaving | allowed (protects running games) |
| reconnects (`hello` with token) | allowed |
Refusing new games sheds load by itself: finished games are not replaced, so
the load falls and the server recovers without intervention.
### Telling the players
Every connected client gets a `server_status` message right after `welcome`
and whenever the level or `accepting_games` changes:
```json
{"type": "server_status", "load": "overloaded", "accepting_games": false}
```
Changes come from the load monitor (`on_change`) and from lobbies
(`on_status_change` when a game starts, finishes or is abandoned), and are
de-duplicated so an unchanged status is not re-sent.
### Health endpoint
`GET /health` (public: `https://game.marvinkrausser.com/wizard/api/health`):
```json
{
"status": "ok",
"load": "normal", "loop_lag": 0.0003, "cpu": 0.012, "throttled": 0.0,
"accepting_games": true,
"lobbies": 4, "active_games": 3, "sessions": 9, "connections": 7,
"ai_available": true,
"ai": {"connections": 1, "pending": 0, "paused": false}
}
```
`status` **stays "ok" under load** on purpose: the Docker health check must
never mark an overloaded server unhealthy, because a restart would end every
running game.
## Layer 4 — AI cost bounds
### In the server (`ai_bridge.py`)
| Mechanism | Setting (default) | Behaviour |
|-----------|-------------------|-----------|
| Deadline | `AI_DECISION_TIMEOUT` (5 s) | sent as `timeout` in every request; no answer in time → random move |
| In-flight cap | `AI_MAX_INFLIGHT` (64) | if that many requests already wait on an AI connection, the move is played randomly without asking. Keep it ≥ `MAX_ACTIVE_GAMES`: every game has at most one AI deciding at a time |
| Circuit breaker | `AI_BREAKER_FAILURES` (5), `AI_BREAKER_COOLDOWN` (30 s) | after 5 **real failures** in a row (timeouts, lost connection, malformed/illegal answers, unexpected errors) the AI is not asked for 30 s; games play random AI moves instead of waiting 5 s per move |
What counts as a failure matters: the breaker is for a *broken or hanging* AI
service. Deliberate load shedding — the in-flight cap, or the AI answering
`overloaded` / `expired` — is **not** a failure: the service is healthy and
answered immediately, and pausing it would only make play worse.
### In the AI service (`wizard_ai/wizard_ai/service/client.py`)
```mermaid
flowchart TD
R[decide request] --> Q{"in progress ≥<br/>AI_MAX_QUEUE?"}
Q -- yes --> E1["error: overloaded"]
Q -- no --> W[wait for a free worker<br/>AI_WORKERS]
W --> X{"server timeout<br/>already passed?"}
X -- yes --> E2["error: expired"]
X -- no --> D{"≥ AI_DEGRADE_QUEUE others waiting,<br/>or primary agent slower than<br/>half the remaining time?"}
D -- yes --> H[heuristic agent]
D -- no --> M[primary agent / model]
H --> OK["action + agent name"]
M --> OK
```
| Setting (default) | Meaning |
|-------------------|---------|
| `AI_WORKERS` (1) | decisions computed at the same time; agents must be thread safe if > 1 |
| `AI_MAX_QUEUE` (64) | requests in progress (waiting + computing) before refusing with `overloaded` |
| `AI_DEGRADE_QUEUE` (8) | from this many other waiting requests, the heuristic decides instead of the model |
The service also keeps a smoothed average of the primary agent's decision
time; if it exceeds half of the time left before the server's deadline, the
heuristic answers instead. The reply's `agent` field says who decided (the
server logs it at debug level). With the heuristic as primary agent (today's
default without a trained model) there is nothing cheaper to fall back to, so
degradation only matters once a real model is deployed.
## What players see
| Situation | Start page | Lobby | Running game |
|-----------|-----------|-------|--------------|
| Normal | — | — | — |
| Not accepting games (overloaded or game limit) | notice "The server is busy…", *Create lobby* disabled, *Join* works | notice, *Start game* shows "Server busy" (disabled) | unchanged |
| Connection limit reached | red banner "The server is full right now. Retrying in a moment…", retries after 15 s, 30 s, 60 s … | — | — |
| Message rate exceeded | toast "Too many messages, slow down." | same | same |
| AI overloaded / broken | — | — | AI players make weaker (heuristic or random) moves |
Implemented in `useWizard.ts` (`serverStatus`, connection status `busy`),
`App.tsx`, `HomeScreen.tsx` and `LobbyScreen.tsx`.
## Configuration reference
All settings with their defaults. Server settings are read in
`wizard_server/app/config.py`, AI settings in `wizard_ai/wizard_ai/config.py`.
Settings marked *compose* are already wired through `compose.yaml` / `.env`;
the others can be added to the `environment:` section of the service.
| Variable | Component | Default | Wired in compose |
|----------|-----------|---------|------------------|
| `SERVER_CPUS` / `SERVER_MEMORY` | Docker | 1.0 / 512M | yes |
| `AI_CPUS` / `AI_MEMORY` | Docker | 1.0 / 1G | yes |
| `WEBSITE_CPUS` / `WEBSITE_MEMORY` | Docker | 0.25 / 64M | yes |
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | 500 / 10 | yes |
| `MESSAGE_RATE` / `MESSAGE_BURST` | server | 10 / 30 | no |
| `MAX_RATE_VIOLATIONS` | server | 50 | no |
| `LOBBY_CREATE_RATE` | server | 10 per minute | no |
| `MAX_LOBBIES` | server | 1000 | no |
| `MAX_ACTIVE_GAMES` | server | 50 | yes |
| `LOAD_SAMPLE_INTERVAL` | server | 1.0 s | no |
| `LOAD_LAG_BUSY` / `LOAD_LAG_OVERLOAD` / `LOAD_LAG_SEVERE` | server | 0.1 / 0.25 / 1.0 s | no |
| `LOAD_CPU_BUSY` / `LOAD_CPU_OVERLOAD` | server | 0.7 / 0.9 | no |
| `LOAD_THROTTLE_OVERLOAD` | server | 0.5 | no |
| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | server | 3 / 10 | no |
| `LOAD_FORCE_LEVEL` | server | empty | no |
| `AI_DECISION_TIMEOUT` | server | 5 s | no |
| `AI_MAX_INFLIGHT` | server | 64 | no |
| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | server | 5 / 30 s | no |
| `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | 1 / 64 / 8 | yes |
## Measurements
Measured on a development machine (uvicorn, heuristic AI, normal pacing:
`AI_MOVE_DELAY` 0.8 s, one scripted human per game taking 1.5 s per move,
4 players per game):
| Running games | Server CPU | Server memory | AI service CPU | AI memory |
|---------------|-----------|---------------|----------------|-----------|
| 0 | ~0 % | ~55–80 MB | ~0 % | ~30 MB |
| 50 | 0.8 % of a core | ~88 MB | 0.5 % | ~30 MB |
| 100 | 1.5 % of a core | ~99 MB | 0.9 % | ~30 MB |
Overload test: server limited to **15 % of one CPU**
(`systemd-run --user --scope -p CPUQuota=15%`), 30 games playing at full speed
(no delays, instant human moves):
| Time | Event |
|------|-------|
| 0 s | load starts |
| ~1 s | `busy` |
| ~3 s | `overloaded` (CPU at 100 % of quota, throttled 100 %) → `create_lobby` refused with `busy` |
| … | running games keep playing (534 moves made while overloaded) |
| ~14 s | games that finished could not restart → load gone → back to `normal` |
What this means:
- With realistic pacing a game costs almost nothing; **1 CPU is far more than
50 games need**. `MAX_ACTIVE_GAMES=50` is a conservative safety cap, not a
CPU requirement — raising it to a few hundred is fine with the heuristic AI.
- Overload only happens with abnormal traffic (scripts, floods) or a heavy AI
model. That is exactly what the limits are for.
- A neural-network model will be the most expensive part. Measure its
decision time and CPU with `python -m wizard_ai evaluate` and `docker stats`
before raising the caps.
## Tuning guide
| Symptom | Adjust |
|---------|--------|
| "Server busy" although `/health` shows low `cpu` and `load: normal` | the game limit is reached: raise `MAX_ACTIVE_GAMES` (and `AI_MAX_INFLIGHT` to at least the same value) |
| "Server busy" with high `cpu` / `throttled` | give the server more CPU (`SERVER_CPUS`) — or accept it: the server is protecting itself |
| Friends behind one router get "server full" | raise `MAX_CONNECTIONS_PER_IP` |
| Busy/overloaded flips too eagerly on a slow machine | raise `LOAD_LAG_*` thresholds or `LOAD_OVERLOAD_SAMPLES` |
| AI plays weaker under load (logs: "Busy … using heuristic agent") | raise `AI_CPUS` and `AI_WORKERS` (thread-safe agent!) or use a smaller model; `AI_DEGRADE_QUEUE` sets how early it degrades |
| Server logs "AI failed 5 times in a row, pausing AI requests" | the AI service hangs or crashes: check `docker compose logs wizard_ai` |
| Server logs "… AI requests already waiting" | more simultaneous AI turns than `AI_MAX_INFLIGHT`: raise it to ≥ `MAX_ACTIVE_GAMES` |
| Container restarts, `docker inspect` shows `OOMKilled: true` | raise `SERVER_MEMORY` / `AI_MEMORY` |
## Monitoring
- `curl -s https://game.marvinkrausser.com/wizard/api/health` — load, CPU,
games, connections, AI state (see above).
- `docker stats` — live CPU and memory per container against the caps.
- Log lines to watch (`docker compose logs -f wizard_server wizard_ai`):
| Log line | Meaning |
|----------|---------|
| `Load level normal -> busy` / `busy -> overloaded` / `-> normal` | load level changes (with lag, CPU, throttling) |
| `Refusing connection from <ip>: connection limit reached` | connection limits hit |
| `Closing connection from <ip>: message flood` | a client was cut off |
| `AI failed N times in a row, pausing AI requests for 30s` | circuit breaker opened |
| `Playing random move for player N: …` | an AI move fell back to random (reason included) |
| AI: `Overloaded: N requests in progress` / `Dropping expired request` / `Busy (N waiting): using heuristic agent` | the AI service shedding load |
## Testing it yourself
Automated tests:
- `wizard_server/tests/test_limits.py` — token bucket, limiters, load levels
and hysteresis, cgroup parsing, AI circuit breaker and in-flight cap.
- `wizard_server/tests/test_protection.py` — floods, connection limits, lobby
creation rate, forced overload, `MAX_ACTIVE_GAMES`, a game continuing while
overloaded, `server_status` messages.
- `wizard_ai/tests/test_ai.py` — queue full, expired requests, degradation.
Manual checks:
```bash
# Refusal UI without real load
cd wizard_server && LOAD_FORCE_LEVEL=overloaded AI_SHARED_SECRET=dev uvicorn app.main:app
# Game limit: start one game, then a second lobby's "Start game" is refused
cd wizard_server && MAX_ACTIVE_GAMES=1 AI_SHARED_SECRET=dev uvicorn app.main:app
# Real CPU exhaustion on Linux: cap the server at 15 % of a CPU,
# then run many games without delays (e.g. a small websocket script)
cd wizard_server && AI_MOVE_DELAY=0 TRICK_PAUSE=0 MAX_CONNECTIONS_PER_IP=200 \
MESSAGE_RATE=1000 MESSAGE_BURST=1000 AI_SHARED_SECRET=dev \
systemd-run --user --scope -p CPUQuota=15% uvicorn app.main:app
```
## Known limitations
- A running game whose humans all disconnected keeps its `MAX_ACTIVE_GAMES`
slot until their sessions expire (`SESSION_TTL`, 30 min), because they may
reconnect. It costs no CPU meanwhile.
- The limits are per server process. That is fine because the server must run
as a single process anyway (all state is in memory).
- Players behind one IP share `MAX_CONNECTIONS_PER_IP` and `LOBBY_CREATE_RATE`.
- There is no waiting queue: players have to retry when the server is busy.
docs/README.md
+2 -1
View File
@@ -11,11 +11,12 @@ to be read top to bottom the first time and used as a reference afterwards.
| 1 | [Architecture overview](01-architecture.md) | The four components, how a move travels through the system, repository layout, key design decisions | | 1 | [Architecture overview](01-architecture.md) | The four components, how a move travels through the system, repository layout, key design decisions |
| 2 | [Game rules as implemented](02-game-rules.md) | Every rule of Wizard as the code implements it, with the exact function that enforces it | | 2 | [Game rules as implemented](02-game-rules.md) | Every rule of Wizard as the code implements it, with the exact function that enforces it |
| 3 | [Game engine (`wizard_game`)](03-game-engine.md) | The rules package: cards, the `WizardGame` state machine, observations, actions, events | | 3 | [Game engine (`wizard_game`)](03-game-engine.md) | The rules package: cards, the `WizardGame` state machine, observations, actions, events |
| 4 | [Game server (`wizard_server/app`)](04-server.md) | Sessions, reconnects, lobbies, running games, AI turns, broadcasting, configuration | | 4 | [Game server (`wizard_server/app`)](04-server.md) | Sessions, reconnects, lobbies, running games, AI turns, broadcasting, limits (summary), configuration |
| 5 | [Protocol reference](05-protocol.md) | Every websocket message between browser ↔ server and server ↔ AI, with examples | | 5 | [Protocol reference](05-protocol.md) | Every websocket message between browser ↔ server and server ↔ AI, with examples |
| 6 | [Frontend (`wizard_website`)](06-frontend.md) | The React app: connection hook, screens, components, styling, build | | 6 | [Frontend (`wizard_website`)](06-frontend.md) | The React app: connection hook, screens, components, styling, build |
| 7 | [AI player (`wizard_ai`)](07-ai.md) | Agents, feature encoding, policy interface, training, evaluation, production service — and how to plug in a real model | | 7 | [AI player (`wizard_ai`)](07-ai.md) | Agents, feature encoding, policy interface, training, evaluation, production service — and how to plug in a real model |
| 8 | [Deployment, development & operations](08-operations.md) | Docker, networks, Caddy, environment variables, local setup, tests, troubleshooting, how-to recipes | | 8 | [Deployment, development & operations](08-operations.md) | Docker, networks, Caddy, environment variables, local setup, tests, troubleshooting, how-to recipes |
| 9 | [Limits, load control and performance](09-limits-and-performance.md) | Resource caps, abuse limits, overload detection, what happens when compute runs out, AI cost bounds, measurements, tuning and monitoring |
## The system in one paragraph ## The system in one paragraph
wizard_ai/README.md
+6 -2
View File
@@ -16,12 +16,16 @@ provides everything around it.
`Policy.update`, with checkpointing `Policy.update`, with checkpointing
- `training/evaluate.py` — play agents against each other (mean score, win rate) - `training/evaluate.py` — play agents against each other (mean score, win rate)
- `service/client.py` — production: connects to `ws://wizard_server:8083/internal/ai`, - `service/client.py` — production: connects to `ws://wizard_server:8083/internal/ai`,
answers `decide` requests, reconnects automatically answers `decide` requests, reconnects automatically. Work is bounded: `AI_WORKERS` (1)
parallel decisions, at most `AI_MAX_QUEUE` (64) requests in progress (more are answered
"overloaded"), expired requests are dropped, and from `AI_DEGRADE_QUEUE` (8) waiting requests —
or when the model is too slow — the heuristic decides instead of the model
(see `../docs/09-limits-and-performance.md`)
```bash ```bash
pip install -r requirements-dev.txt pip install -r requirements-dev.txt
python -m wizard_ai train --episodes 5000 --out models/latest python -m wizard_ai train --episodes 5000 --out models/latest
python -m wizard_ai evaluate --agents model,heuristic,random,random --games 200 python -m wizard_ai evaluate --agents model,heuristic,random,random --games 200
python -m wizard_ai serve # env: SERVER_URL, AI_SHARED_SECRET, AGENT, MODEL_PATH python -m wizard_ai serve # env: SERVER_URL, AI_SHARED_SECRET, AGENT, MODEL_PATH, AI_WORKERS, AI_MAX_QUEUE, AI_DEGRADE_QUEUE
pytest pytest
``` ```
wizard_server/README.md
+11 -3
View File
@@ -8,11 +8,19 @@ FastAPI game server. It manages sessions, lobbies and games, and talks to the AI
- `app/protocol.py` — client message schema - `app/protocol.py` — client message schema
- `app/service.py` — dispatches client messages, reconnects sessions, cleanup - `app/service.py` — dispatches client messages, reconnects sessions, cleanup
- `app/lobby.py` — lobbies, seats, host actions, running games and AI turns - `app/lobby.py` — lobbies, seats, host actions, running games and AI turns
- `app/ai_bridge.py` — request/response protocol with the AI service, random-move fallback - `app/ai_bridge.py` — request/response protocol with the AI service, random-move fallback,
in-flight cap and circuit breaker
- `app/sessions.py` — sessions (token based, survive reloads) and ordered websocket sending - `app/sessions.py` — sessions (token based, survive reloads) and ordered websocket sending
- `app/limits.py` — token bucket, per-IP rate limiter, connection limiter (abuse protection)
- `app/load.py` — load monitor (event loop lag, CPU, cgroup throttling) with hysteresis; the
service refuses new lobbies/games while overloaded or at `MAX_ACTIVE_GAMES`
Configuration (environment): `AI_SHARED_SECRET`, `AI_DECISION_TIMEOUT` (5s), `AI_MOVE_DELAY` (0.8s), Configuration (environment, all in `app/config.py`): `AI_SHARED_SECRET`, `AI_DECISION_TIMEOUT` (5s),
`TRICK_PAUSE` (1.8s), `SESSION_TTL` (1800s), `MAX_LOBBIES` (1000), `ALLOWED_ORIGINS`, `LOG_LEVEL`. `AI_MOVE_DELAY` (0.8s), `TRICK_PAUSE` (1.8s), `SESSION_TTL` (1800s), `MAX_LOBBIES` (1000),
`ALLOWED_ORIGINS`, `LOG_LEVEL`, and the limits: `MAX_CONNECTIONS` (500), `MAX_CONNECTIONS_PER_IP` (10),
`MESSAGE_RATE`/`MESSAGE_BURST` (10/30), `LOBBY_CREATE_RATE` (10/min), `MAX_ACTIVE_GAMES` (50),
`LOAD_*` thresholds, `AI_MAX_INFLIGHT` (64), `AI_BREAKER_FAILURES`/`AI_BREAKER_COOLDOWN` (5/30s).
See `../docs/04-server.md` and `../docs/09-limits-and-performance.md`.
```bash ```bash
pip install -r requirements-dev.txt pip install -r requirements-dev.txt
wizard_website/README.md
+6 -3
View File
@@ -18,9 +18,12 @@ development, or `VITE_WS_URL` at build time to use an absolute websocket URL ins
## Structure ## Structure
- `src/api/` — protocol types and the `useWizard` hook (websocket connection, - `src/api/` — protocol types and the `useWizard` hook (websocket connection,
reconnect with a per-tab session token, game state) reconnect with a per-tab session token, game state, server load status; slow
retries when the server is full)
- `src/components/` — reusable UI (button, playing card, modal, toasts) - `src/components/` — reusable UI (button, playing card, modal, toasts)
- `src/screens/HomeScreen` — enter a name, create or join a lobby - `src/screens/HomeScreen` — enter a name, create or join a lobby (Create is disabled with a
- `src/screens/LobbyScreen` — lobby code, players, AI players, rule options, start notice while the server is busy)
- `src/screens/LobbyScreen` — lobby code, players, AI players, rule options, start (disabled
with a notice while the server is busy)
- `src/screens/GameScreen` — the table: seats, trick, hand, bidding, trump choice, - `src/screens/GameScreen` — the table: seats, trick, hand, bidding, trump choice,
scoreboard, round summary, game over scoreboard, round summary, game over