Document limits, load control and performance
- New docs/09-limits-and-performance.md: principles, all protection layers, what players see, configuration reference, measurements, tuning guide, monitoring and how to test - Server chapter summarises the limits and links to chapter 9; circuit breaker description and AI_MAX_INFLIGHT default corrected - Architecture: new files in the layout, design decisions for load handling; docs index lists chapter 9 - Protocol, AI and operations chapters link to chapter 9; test counts - Root and component READMEs describe the limits and settings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+12
-6
@@ -5,7 +5,7 @@
|
||||
| Component | Folder | Language / stack | Runs as | Responsibility |
|
||||
|-----------|--------|------------------|---------|----------------|
|
||||
| Website | `wizard_website/` | React 19, TypeScript, Vite, CSS modules | static files served by nginx (container port 8082) | UI: start page, lobby, game table |
|
||||
| Game server | `wizard_server/app/` | Python 3.13, FastAPI, uvicorn | one process (container port 8083) | sessions, lobbies, running games, asking the AI for moves |
|
||||
| Game server | `wizard_server/app/` | Python 3.13, FastAPI, uvicorn | one process (container port 8083) | sessions, lobbies, running games, asking the AI for moves, limits and load control |
|
||||
| Game engine | `wizard_server/game_logic/` (package `wizard_game`) | pure Python, no dependencies | library, installed into server and AI images | the rules of Wizard |
|
||||
| AI service | `wizard_ai/` | Python, `websockets` | one process, no listening port | chooses moves for AI seats; training tools |
|
||||
|
||||
@@ -89,8 +89,8 @@ Important properties of this flow:
|
||||
|
||||
```
|
||||
.
|
||||
├── compose.yaml # all three containers + networks
|
||||
├── .env.example # template for .env (AI_SHARED_SECRET, …)
|
||||
├── compose.yaml # all three containers, networks, CPU/memory caps
|
||||
├── .env.example # template for .env (secret, resource limits, capacity)
|
||||
├── README.md # short project readme
|
||||
├── docs/ # this documentation
|
||||
├── instructions.txt # original task description
|
||||
@@ -107,8 +107,11 @@ Important properties of this flow:
|
||||
│ │ ├── service.py # WizardService: dispatches messages, sessions, cleanup
|
||||
│ │ ├── sessions.py # Session, Connection (ordered send queue), SessionRegistry
|
||||
│ │ ├── lobby.py # Lobby, Seat, LobbyManager, AI turn loop
|
||||
│ │ └── ai_bridge.py # request/response with the AI service + fallback
|
||||
│ ├── tests/test_server.py # websocket-level tests of the server
|
||||
│ │ ├── ai_bridge.py # request/response with the AI service, fallback, circuit breaker
|
||||
│ │ ├── limits.py # TokenBucket, rate and connection limiters
|
||||
│ │ └── load.py # LoadMonitor: detects overload (lag, CPU, cgroup throttling)
|
||||
│ ├── tests/ # conftest.py/helpers.py (fixtures), test_server.py,
|
||||
│ │ # test_protection.py (limits/overload), test_limits.py (units)
|
||||
│ └── game_logic/ # installable package "wizard-game"
|
||||
│ ├── pyproject.toml
|
||||
│ ├── wizard_game/
|
||||
@@ -130,7 +133,7 @@ Important properties of this flow:
|
||||
│ │ ├── agents/ # Agent interface, random, heuristic, model agent
|
||||
│ │ ├── model/ # encoding.py (features/actions), policy.py (model interface)
|
||||
│ │ ├── training/ # env.py, self_play.py, evaluate.py
|
||||
│ │ └── service/client.py # production websocket client
|
||||
│ │ └── service/client.py # production websocket client (workers, queue, degradation)
|
||||
│ └── tests/test_ai.py
|
||||
│
|
||||
└── wizard_website/
|
||||
@@ -158,3 +161,6 @@ Important properties of this flow:
|
||||
| Session token in `sessionStorage` | Survives page reloads (reconnect to the same seat) but is per tab, so you can test several players in several tabs. |
|
||||
| Per-connection send queue | Websocket messages are always sent in order and never concurrently (see `sessions.Connection`). |
|
||||
| Relative asset paths + relative API URL in the frontend | The app works under any path prefix (`/wizard/website/` in production, `/` in development) without rebuilding. |
|
||||
| Refuse new games under load, never interrupt running ones | Players in a game are never punished for load; creating/starting games is what adds load, so that is what gets refused ("server busy"). Refusing sheds load by itself. See [chapter 9](09-limits-and-performance.md). |
|
||||
| Hard Docker caps plus software limits | The caps guarantee the host is protected even if a software limit fails; the software limits make overload visible and graceful instead of just slow. |
|
||||
| Health check stays "ok" under load | Docker must never restart an overloaded server — that would end every running game. |
|
||||
|
||||
+26
-88
@@ -256,97 +256,35 @@ The AI service keeps a websocket open to `/internal/ai`. `AIBridge`:
|
||||
server has already given up on;
|
||||
- **in-flight cap**: if `AI_MAX_INFLIGHT` requests are already waiting on a
|
||||
connection, the move is played by the fallback without asking;
|
||||
- **circuit breaker**: after `AI_BREAKER_FAILURES` failures in a row (timeouts,
|
||||
errors, illegal answers) the AI is not asked for `AI_BREAKER_COOLDOWN`
|
||||
seconds, so a struggling AI service is not buried under more requests.
|
||||
- **circuit breaker**: after `AI_BREAKER_FAILURES` real failures in a row
|
||||
(timeouts, lost connection, malformed/illegal answers, unexpected errors) the
|
||||
AI is not asked for `AI_BREAKER_COOLDOWN` seconds, so games don't wait
|
||||
`AI_DECISION_TIMEOUT` for every move of a hanging AI. Load shedding (the
|
||||
in-flight cap, AI answers `overloaded` / `expired`) does not count.
|
||||
|
||||
## Limits and overload handling
|
||||
|
||||
The server protects the host and its players in three layers. The guiding rule
|
||||
is: **running games always continue; new work is refused when there is no
|
||||
capacity; abuse is rejected immediately.**
|
||||
The complete description — all layers, thresholds, what players see, tuning,
|
||||
monitoring and measurements — is in
|
||||
[chapter 9: Limits, load control and performance](09-limits-and-performance.md).
|
||||
In short, for the server:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
C[new websocket] --> L1{"connection limit<br/>total / per IP"}
|
||||
L1 -- over --> R1["error busy + close 1013"]
|
||||
L1 -- ok --> M[message]
|
||||
M --> L2{"message rate<br/>token bucket"}
|
||||
L2 -- over --> R2["dropped + error rate_limited<br/>(close 1008 when flooding)"]
|
||||
L2 -- ok --> D{"create_lobby /<br/>start_game?"}
|
||||
D -- no --> OK[handled normally]
|
||||
D -- yes --> L3{"accepting_games?<br/>not overloaded and<br/>running games < MAX_ACTIVE_GAMES"}
|
||||
L3 -- no --> R3["error busy"]
|
||||
L3 -- yes --> OK
|
||||
```
|
||||
|
||||
### 1. Hard limits (Docker and uvicorn)
|
||||
|
||||
- `compose.yaml` caps every container (`deploy.resources.limits` + `pids_limit`):
|
||||
server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB by default,
|
||||
overridable in `.env` (`SERVER_CPUS`, `SERVER_MEMORY`, …).
|
||||
- uvicorn (`wizard_server/Dockerfile`): `--ws-max-size 65536` (bigger messages
|
||||
close the connection with 1009), `--ws-max-queue 16`,
|
||||
`--limit-concurrency 1000` (HTTP 503 beyond that — a last resort above the
|
||||
app's own connection limit).
|
||||
|
||||
### 2. Abuse limits (`limits.py`, applied in `WizardService`)
|
||||
|
||||
| Limit | Default | When exceeded |
|
||||
|-------|---------|---------------|
|
||||
| open connections total / per client IP | 500 / 10 | `error` (code `busy`), then close **1013** "Try Again Later" |
|
||||
| messages per connection (token bucket) | 10/s, burst 30 | message dropped, `error` (code `rate_limited`, not repeated for every message); after 50 drops the connection is closed with 1008 |
|
||||
| lobbies created per client IP | 10 per minute | `error` (code `rate_limited`) |
|
||||
|
||||
The client IP is `websocket.client.host`. Behind Caddy, uvicorn's
|
||||
`--proxy-headers` sets it from `X-Forwarded-For` (Caddy does not pass on
|
||||
client-supplied values, so it cannot be spoofed).
|
||||
|
||||
### 3. Load monitor and admission control (`load.py`)
|
||||
|
||||
`LoadMonitor.run()` (started in the app lifespan) samples once per second:
|
||||
|
||||
- **event loop lag** — how late the loop wakes up; the most direct "players
|
||||
notice slowness" signal;
|
||||
- **CPU** — process CPU time relative to the container's CPU quota (read from
|
||||
the process's own cgroup, `/proc/self/cgroup` → `…/cpu.max`), at most one
|
||||
core because game logic runs on one core;
|
||||
- **throttling** — share of CPU periods in which the container hit its limit
|
||||
(`cpu.stat`: `nr_throttled / nr_periods`).
|
||||
|
||||
Each sample is classified (`classify`) and `next_level` derives the level with
|
||||
hysteresis:
|
||||
|
||||
| Level | Entered when | Left when |
|
||||
|-------|--------------|-----------|
|
||||
| `busy` | one sample with lag ≥ 0.1 s or CPU ≥ 70 % | 10 normal samples in a row |
|
||||
| `overloaded` | 3 samples in a row with lag ≥ 0.25 s, CPU ≥ 90 % or throttled ≥ 50 % — or **one** sample with lag ≥ 1 s | 10 samples in a row that are not overloaded |
|
||||
|
||||
`busy` is only informational. While **overloaded**, or while
|
||||
`MAX_ACTIVE_GAMES` (default 50) games are running, `accepting_games` is false:
|
||||
|
||||
| Action | Refused when not accepting games |
|
||||
|--------|----------------------------------|
|
||||
| `create_lobby` | yes (`error` code `busy`) |
|
||||
| `start_game` | yes (`error` code `busy`) |
|
||||
| `join_lobby`, moves, `return_to_lobby`, reconnects | never — they are cheap and keep running games alive |
|
||||
|
||||
Refusing new games sheds load by itself: finished games are not replaced, so
|
||||
the load drops and the server recovers.
|
||||
|
||||
Every connected client receives a `server_status` message after `welcome` and
|
||||
whenever the level or `accepting_games` changes (the website then disables
|
||||
"Create lobby" / "Start game" and shows a notice). `/health` reports `load`,
|
||||
`loop_lag`, `cpu`, `throttled`, `accepting_games`, `active_games`,
|
||||
`connections` and AI bridge stats, but its `status` **stays "ok"** under load:
|
||||
an overloaded server must never be restarted by Docker, which would end all
|
||||
running games.
|
||||
|
||||
Measured on a server capped at 15 % of a CPU with 30 games played at full
|
||||
speed: overloaded after ~3 s, new lobbies refused, running games kept going,
|
||||
back to normal ~10 s after the load dropped.
|
||||
|
||||
For manual testing, `LOAD_FORCE_LEVEL=overloaded` pins the level.
|
||||
- **Rule**: running games always continue; new lobbies/games are refused when
|
||||
there is no capacity; abuse is rejected immediately.
|
||||
- **Abuse limits** (`limits.py`, applied in `WizardService.handle_client`):
|
||||
connections in total and per IP (refused with `error` code `busy` + close
|
||||
1013), a token bucket per connection for messages (`rate_limited`, close
|
||||
1008 on floods), lobby creation per IP.
|
||||
- **Load monitor** (`load.py`, `LoadMonitor.run()` in the app lifespan):
|
||||
samples event loop lag, CPU against the container quota, and cgroup
|
||||
throttling every second; levels `normal` / `busy` / `overloaded` with
|
||||
hysteresis.
|
||||
- **Admission control** (`WizardService.accepting_games`): `create_lobby` and
|
||||
`start_game` are refused while overloaded or while `MAX_ACTIVE_GAMES` games
|
||||
run; everything else keeps working. Changes are broadcast to all clients as
|
||||
`server_status`.
|
||||
- `/health` reports the load but its `status` stays "ok", so Docker never
|
||||
restarts an overloaded server (which would end all games).
|
||||
|
||||
## Configuration (`config.py`)
|
||||
|
||||
@@ -373,7 +311,7 @@ All values are read from environment variables when the module is imported.
|
||||
| `LOAD_THROTTLE_OVERLOAD` | 0.5 | throttled share of CPU periods |
|
||||
| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | 3 / 10 | hysteresis |
|
||||
| `LOAD_FORCE_LEVEL` | empty | pin the level (testing) |
|
||||
| `AI_MAX_INFLIGHT` | 32 | waiting AI requests per AI connection |
|
||||
| `AI_MAX_INFLIGHT` | 64 | waiting AI requests per AI connection (keep ≥ `MAX_ACTIVE_GAMES`) |
|
||||
| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | 5 / 30 s | AI circuit breaker |
|
||||
|
||||
## Housekeeping
|
||||
|
||||
+4
-3
@@ -192,8 +192,9 @@ Rules:
|
||||
- No answer within `AI_DECISION_TIMEOUT` (5 s) → random move. The AI service
|
||||
drops requests whose `timeout` passed before it could start them (`expired`).
|
||||
- The server stops asking an AI connection that already has `AI_MAX_INFLIGHT`
|
||||
unanswered requests, and pauses the AI after repeated failures (circuit
|
||||
breaker); see "Limits and overload handling" in
|
||||
[the server chapter](04-server.md).
|
||||
unanswered requests, and pauses the AI after repeated real failures (circuit
|
||||
breaker). `overloaded` / `expired` answers are load shedding, not failures,
|
||||
and do not trip the breaker. See
|
||||
"Layer 4" in [chapter 9](09-limits-and-performance.md).
|
||||
- Several AI service instances may connect; the server distributes requests
|
||||
round robin.
|
||||
|
||||
+3
-1
@@ -268,7 +268,9 @@ count as a win for everyone tied). Seeded for fair comparisons.
|
||||
primary agent already is the heuristic. The reply's `agent` field says which
|
||||
one decided;
|
||||
- `OMP_NUM_THREADS` / `MKL_NUM_THREADS` are set to `AI_WORKERS` in
|
||||
`compose.yaml`, so numeric libraries don't start a thread per host core;
|
||||
`compose.yaml`, so numeric libraries don't start a thread per host core
|
||||
(the whole picture, including the server side, is in
|
||||
[chapter 9](09-limits-and-performance.md));
|
||||
- errors while deciding are reported back as `error` (the server then plays a
|
||||
random move) instead of crashing;
|
||||
- reconnects forever with exponential backoff (1 s … 30 s); logs a hint if the
|
||||
|
||||
@@ -51,8 +51,8 @@ not read by anything.
|
||||
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | no (500 / 10) | open websocket connections |
|
||||
| `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | no (1 / 64 / 8) | parallel decisions, waiting requests before refusing, queue depth from which the heuristic is used |
|
||||
|
||||
How the limits work together is described in "Limits and overload handling"
|
||||
in [the server chapter](04-server.md). Further server settings
|
||||
How the limits work together, the full list of settings, measurements and a
|
||||
tuning guide are in [chapter 9](09-limits-and-performance.md). Further server settings
|
||||
(`AI_MOVE_DELAY`, `TRICK_PAUSE`, load thresholds, …, see
|
||||
"Configuration" in [the server chapter](04-server.md)) can be added to
|
||||
the `environment:` section of `wizard_server` in `compose.yaml`.
|
||||
@@ -155,7 +155,7 @@ cd wizard_website && npm run lint && npm run build
|
||||
| Suite | Count | Covers |
|
||||
|-------|-------|--------|
|
||||
| game engine | 52 | rules, full random games for 3–6 players, invariants, JSON |
|
||||
| server | 32 | websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels |
|
||||
| server | 33 | websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels |
|
||||
| AI | 17 | encoding, agents, env, trainer, checkpoints, service limits |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -0,0 +1,373 @@
|
||||
# 9. Limits, load control and performance
|
||||
|
||||
This chapter explains how Wizard keeps its resource use bounded, how it
|
||||
protects itself against abuse, and what happens when it runs out of compute.
|
||||
It covers all components, because the protection works in layers from Docker
|
||||
down to the browser.
|
||||
|
||||
## Principles
|
||||
|
||||
1. **Running games always continue.** A game in progress is never stopped,
|
||||
slowed down on purpose or refused a move because of load.
|
||||
2. **New work is refused when there is no capacity.** Creating lobbies and
|
||||
starting games is what adds load, so that is what gets refused — with a
|
||||
clear "server busy" message, not a timeout. (A waiting queue was considered
|
||||
and rejected as not worth the complexity.)
|
||||
3. **Abuse is rejected immediately.** Floods and connection storms are cut off
|
||||
at the door, never queued.
|
||||
4. **The host is protected no matter what.** Hard Docker caps guarantee that
|
||||
Wizard can never take CPU or memory from the other services on the machine,
|
||||
even if every software limit fails.
|
||||
5. **Degrade, don't fail.** An overloaded or broken AI makes weaker moves (the
|
||||
heuristic, or a random move) instead of stopping games.
|
||||
|
||||
## Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph docker["Layer 1: Docker caps (compose.yaml)"]
|
||||
direction TB
|
||||
subgraph server["wizard_server — 1 CPU / 512 MB"]
|
||||
U["uvicorn: max message 64 KB,<br/>max 1000 connections"]
|
||||
A["Layer 2: abuse limits<br/>connections, message rate,<br/>lobby creation (limits.py)"]
|
||||
L["Layer 3: load monitor +<br/>admission control (load.py)"]
|
||||
B["AI bridge: in-flight cap,<br/>circuit breaker (ai_bridge.py)"]
|
||||
end
|
||||
subgraph ai["wizard_ai — 1 CPU / 1 GB"]
|
||||
Q["Layer 4: workers, queue,<br/>deadlines, degradation<br/>(service/client.py)"]
|
||||
end
|
||||
W["wizard_website — 0.25 CPU / 64 MB"]
|
||||
end
|
||||
Browser -->|websocket| U --> A --> L
|
||||
L -->|AI turns| B -->|decide| Q
|
||||
L -. "server_status" .-> Browser
|
||||
```
|
||||
|
||||
## Layer 1 — Docker caps
|
||||
|
||||
Set in `compose.yaml` from `.env` (defaults shown):
|
||||
|
||||
| Container | CPU | Memory | Processes |
|
||||
|-----------|-----|--------|-----------|
|
||||
| `wizard_server` | `SERVER_CPUS=1.0` | `SERVER_MEMORY=512M` | 256 |
|
||||
| `wizard_ai` | `AI_CPUS=1.0` | `AI_MEMORY=1G` | 256 |
|
||||
| `wizard_website` | `WEBSITE_CPUS=0.25` | `WEBSITE_MEMORY=64M` | 64 |
|
||||
|
||||
- **CPU** is a hard quota: when the container uses it up, the kernel
|
||||
throttles it. The server notices this (see layer 3) and refuses new games.
|
||||
- **Memory** is a hard limit: a container exceeding it is killed by the kernel
|
||||
and restarted by Docker (`restart: unless-stopped`). For the server this
|
||||
would end all games (state is in memory), so the limit has lots of headroom:
|
||||
the server uses ~80 MB idle and ~100 MB with 100 games (see
|
||||
[measurements](#measurements)).
|
||||
- **Processes** (`pids_limit`) stop fork bombs / thread explosions.
|
||||
- The AI container sets `OMP_NUM_THREADS` / `MKL_NUM_THREADS` to `AI_WORKERS`,
|
||||
so numeric libraries (numpy, torch) don't start one thread per *host* core
|
||||
inside a 1-CPU container.
|
||||
|
||||
uvicorn (`wizard_server/Dockerfile`) adds transport limits:
|
||||
`--ws-max-size 65536` (larger messages close the connection with code 1009;
|
||||
real messages are < 1 KB), `--ws-max-queue 16`, `--limit-concurrency 1000`
|
||||
(HTTP 503 beyond that — a last resort above the app's own connection limit).
|
||||
|
||||
## Layer 2 — abuse limits (server)
|
||||
|
||||
`wizard_server/app/limits.py`, applied in `WizardService.handle_client`
|
||||
(`service.py`):
|
||||
|
||||
| Limit | Setting (default) | When exceeded |
|
||||
|-------|-------------------|---------------|
|
||||
| Open connections in total | `MAX_CONNECTIONS` (500) | `error` code `busy`, close **1013** "Try Again Later" |
|
||||
| Open connections per client IP | `MAX_CONNECTIONS_PER_IP` (10) | same |
|
||||
| Messages per connection | `MESSAGE_RATE` (10/s), `MESSAGE_BURST` (30) | message dropped; `error` code `rate_limited` (only on the 1st and every 10th drop) |
|
||||
| Flooding | `MAX_RATE_VIOLATIONS` (50 drops) | connection closed with **1008** |
|
||||
| Lobbies created per client IP | `LOBBY_CREATE_RATE` (10 per minute) | `error` code `rate_limited` |
|
||||
|
||||
Building blocks:
|
||||
|
||||
- `TokenBucket(rate, burst)` — the classic algorithm: a bucket holds up to
|
||||
`burst` tokens, refills at `rate` per second, each action takes one token.
|
||||
Allows short bursts (a player clicking quickly) but bounds the average.
|
||||
- `KeyedRateLimiter` — one bucket per key (client IP); buckets that refilled
|
||||
completely are forgotten by the cleanup task every 30 s.
|
||||
- `ConnectionLimiter` — counts open connections in total and per IP; released
|
||||
in the `finally` block of `handle_client`, so crashes cannot leak slots.
|
||||
|
||||
**Client IP**: `websocket.client.host`. Behind Caddy, uvicorn's
|
||||
`--proxy-headers` sets it from `X-Forwarded-For`; Caddy ignores
|
||||
client-supplied `X-Forwarded-For` values, so the IP cannot be spoofed. In
|
||||
local development everything comes from `127.0.0.1`.
|
||||
|
||||
Normal play needs about one message per second, so these limits are never
|
||||
noticeable for humans. Several players behind one NAT (e.g. a household)
|
||||
share the per-IP connection limit — raise `MAX_CONNECTIONS_PER_IP` if that is
|
||||
a realistic use case.
|
||||
|
||||
## Layer 3 — load monitor and admission control (server)
|
||||
|
||||
### Detecting overload (`load.py`)
|
||||
|
||||
`LoadMonitor.run()` is started in the app lifespan and takes a sample every
|
||||
`LOAD_SAMPLE_INTERVAL` (1 s):
|
||||
|
||||
| Signal | How it is measured | Why |
|
||||
|--------|--------------------|-----|
|
||||
| **Event loop lag** | the monitor sleeps 1 s and measures how much later it woke up | the most direct measure of "the server reacts slowly"; works everywhere |
|
||||
| **CPU** | process CPU time per second, relative to the container's CPU quota (at most 1 core, because game logic runs on one core) | detects saturation before players notice lag |
|
||||
| **Throttling** | `nr_throttled / nr_periods` from `cpu.stat` | share of scheduling periods in which the container hit its Docker CPU cap |
|
||||
|
||||
CPU quota and throttling come from the process's **own cgroup** (cgroup v2):
|
||||
`/proc/self/cgroup` gives its path, e.g. `0::/` inside a container (→
|
||||
`/sys/fs/cgroup`) or `0::/user.slice/…scope` on a host. Without cgroup files
|
||||
(e.g. macOS) the monitor still works with lag and CPU against one core.
|
||||
|
||||
Each sample is classified (`classify`), and `next_level` turns the recent
|
||||
classifications into the current level with hysteresis:
|
||||
|
||||
| Level | Entered | Left |
|
||||
|-------|---------|------|
|
||||
| `normal` | — | — |
|
||||
| `busy` | one sample with lag ≥ `LOAD_LAG_BUSY` (0.1 s) or CPU ≥ `LOAD_CPU_BUSY` (70 %) | after `LOAD_RECOVER_SAMPLES` (10) normal samples in a row |
|
||||
| `overloaded` | `LOAD_OVERLOAD_SAMPLES` (3) samples in a row with lag ≥ `LOAD_LAG_OVERLOAD` (0.25 s), CPU ≥ `LOAD_CPU_OVERLOAD` (90 %) or throttled ≥ `LOAD_THROTTLE_OVERLOAD` (50 %) — **or one** sample with lag ≥ `LOAD_LAG_SEVERE` (1 s) | after 10 samples in a row that are not overloaded |
|
||||
|
||||
Why hysteresis: without it a server at the edge would flip between states
|
||||
every second, and every flip is broadcast to all players. Why the "severe"
|
||||
rule: when the event loop is extremely late, samples themselves arrive late
|
||||
(one per several seconds), so waiting for three would delay detection a lot.
|
||||
|
||||
`busy` is informational only. `LOAD_FORCE_LEVEL` pins the level for testing.
|
||||
|
||||
### Admission control (`service.py`)
|
||||
|
||||
`WizardService.accepting_games` is true unless the level is `overloaded` or
|
||||
`MAX_ACTIVE_GAMES` (50) games are running (`LobbyManager.active_games()`).
|
||||
|
||||
| Action | While not accepting games |
|
||||
|--------|---------------------------|
|
||||
| `create_lobby` | refused — `error` code `busy` |
|
||||
| `start_game` | refused — `error` code `busy` |
|
||||
| `join_lobby` | allowed (cheap) |
|
||||
| moves in running games, `return_to_lobby`, leaving | allowed (protects running games) |
|
||||
| reconnects (`hello` with token) | allowed |
|
||||
|
||||
Refusing new games sheds load by itself: finished games are not replaced, so
|
||||
the load falls and the server recovers without intervention.
|
||||
|
||||
### Telling the players
|
||||
|
||||
Every connected client gets a `server_status` message right after `welcome`
|
||||
and whenever the level or `accepting_games` changes:
|
||||
|
||||
```json
|
||||
{"type": "server_status", "load": "overloaded", "accepting_games": false}
|
||||
```
|
||||
|
||||
Changes come from the load monitor (`on_change`) and from lobbies
|
||||
(`on_status_change` when a game starts, finishes or is abandoned), and are
|
||||
de-duplicated so an unchanged status is not re-sent.
|
||||
|
||||
### Health endpoint
|
||||
|
||||
`GET /health` (public: `https://game.marvinkrausser.com/wizard/api/health`):
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"load": "normal", "loop_lag": 0.0003, "cpu": 0.012, "throttled": 0.0,
|
||||
"accepting_games": true,
|
||||
"lobbies": 4, "active_games": 3, "sessions": 9, "connections": 7,
|
||||
"ai_available": true,
|
||||
"ai": {"connections": 1, "pending": 0, "paused": false}
|
||||
}
|
||||
```
|
||||
|
||||
`status` **stays "ok" under load** on purpose: the Docker health check must
|
||||
never mark an overloaded server unhealthy, because a restart would end every
|
||||
running game.
|
||||
|
||||
## Layer 4 — AI cost bounds
|
||||
|
||||
### In the server (`ai_bridge.py`)
|
||||
|
||||
| Mechanism | Setting (default) | Behaviour |
|
||||
|-----------|-------------------|-----------|
|
||||
| Deadline | `AI_DECISION_TIMEOUT` (5 s) | sent as `timeout` in every request; no answer in time → random move |
|
||||
| In-flight cap | `AI_MAX_INFLIGHT` (64) | if that many requests already wait on an AI connection, the move is played randomly without asking. Keep it ≥ `MAX_ACTIVE_GAMES`: every game has at most one AI deciding at a time |
|
||||
| Circuit breaker | `AI_BREAKER_FAILURES` (5), `AI_BREAKER_COOLDOWN` (30 s) | after 5 **real failures** in a row (timeouts, lost connection, malformed/illegal answers, unexpected errors) the AI is not asked for 30 s; games play random AI moves instead of waiting 5 s per move |
|
||||
|
||||
What counts as a failure matters: the breaker is for a *broken or hanging* AI
|
||||
service. Deliberate load shedding — the in-flight cap, or the AI answering
|
||||
`overloaded` / `expired` — is **not** a failure: the service is healthy and
|
||||
answered immediately, and pausing it would only make play worse.
|
||||
|
||||
### In the AI service (`wizard_ai/wizard_ai/service/client.py`)
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
R[decide request] --> Q{"in progress ≥<br/>AI_MAX_QUEUE?"}
|
||||
Q -- yes --> E1["error: overloaded"]
|
||||
Q -- no --> W[wait for a free worker<br/>AI_WORKERS]
|
||||
W --> X{"server timeout<br/>already passed?"}
|
||||
X -- yes --> E2["error: expired"]
|
||||
X -- no --> D{"≥ AI_DEGRADE_QUEUE others waiting,<br/>or primary agent slower than<br/>half the remaining time?"}
|
||||
D -- yes --> H[heuristic agent]
|
||||
D -- no --> M[primary agent / model]
|
||||
H --> OK["action + agent name"]
|
||||
M --> OK
|
||||
```
|
||||
|
||||
| Setting (default) | Meaning |
|
||||
|-------------------|---------|
|
||||
| `AI_WORKERS` (1) | decisions computed at the same time; agents must be thread safe if > 1 |
|
||||
| `AI_MAX_QUEUE` (64) | requests in progress (waiting + computing) before refusing with `overloaded` |
|
||||
| `AI_DEGRADE_QUEUE` (8) | from this many other waiting requests, the heuristic decides instead of the model |
|
||||
|
||||
The service also keeps a smoothed average of the primary agent's decision
|
||||
time; if it exceeds half of the time left before the server's deadline, the
|
||||
heuristic answers instead. The reply's `agent` field says who decided (the
|
||||
server logs it at debug level). With the heuristic as primary agent (today's
|
||||
default without a trained model) there is nothing cheaper to fall back to, so
|
||||
degradation only matters once a real model is deployed.
|
||||
|
||||
## What players see
|
||||
|
||||
| Situation | Start page | Lobby | Running game |
|
||||
|-----------|-----------|-------|--------------|
|
||||
| Normal | — | — | — |
|
||||
| Not accepting games (overloaded or game limit) | notice "The server is busy…", *Create lobby* disabled, *Join* works | notice, *Start game* shows "Server busy" (disabled) | unchanged |
|
||||
| Connection limit reached | red banner "The server is full right now. Retrying in a moment…", retries after 15 s, 30 s, 60 s … | — | — |
|
||||
| Message rate exceeded | toast "Too many messages, slow down." | same | same |
|
||||
| AI overloaded / broken | — | — | AI players make weaker (heuristic or random) moves |
|
||||
|
||||
Implemented in `useWizard.ts` (`serverStatus`, connection status `busy`),
|
||||
`App.tsx`, `HomeScreen.tsx` and `LobbyScreen.tsx`.
|
||||
|
||||
## Configuration reference
|
||||
|
||||
All settings with their defaults. Server settings are read in
|
||||
`wizard_server/app/config.py`, AI settings in `wizard_ai/wizard_ai/config.py`.
|
||||
Settings marked *compose* are already wired through `compose.yaml` / `.env`;
|
||||
the others can be added to the `environment:` section of the service.
|
||||
|
||||
| Variable | Component | Default | Wired in compose |
|
||||
|----------|-----------|---------|------------------|
|
||||
| `SERVER_CPUS` / `SERVER_MEMORY` | Docker | 1.0 / 512M | yes |
|
||||
| `AI_CPUS` / `AI_MEMORY` | Docker | 1.0 / 1G | yes |
|
||||
| `WEBSITE_CPUS` / `WEBSITE_MEMORY` | Docker | 0.25 / 64M | yes |
|
||||
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | 500 / 10 | yes |
|
||||
| `MESSAGE_RATE` / `MESSAGE_BURST` | server | 10 / 30 | no |
|
||||
| `MAX_RATE_VIOLATIONS` | server | 50 | no |
|
||||
| `LOBBY_CREATE_RATE` | server | 10 per minute | no |
|
||||
| `MAX_LOBBIES` | server | 1000 | no |
|
||||
| `MAX_ACTIVE_GAMES` | server | 50 | yes |
|
||||
| `LOAD_SAMPLE_INTERVAL` | server | 1.0 s | no |
|
||||
| `LOAD_LAG_BUSY` / `LOAD_LAG_OVERLOAD` / `LOAD_LAG_SEVERE` | server | 0.1 / 0.25 / 1.0 s | no |
|
||||
| `LOAD_CPU_BUSY` / `LOAD_CPU_OVERLOAD` | server | 0.7 / 0.9 | no |
|
||||
| `LOAD_THROTTLE_OVERLOAD` | server | 0.5 | no |
|
||||
| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | server | 3 / 10 | no |
|
||||
| `LOAD_FORCE_LEVEL` | server | empty | no |
|
||||
| `AI_DECISION_TIMEOUT` | server | 5 s | no |
|
||||
| `AI_MAX_INFLIGHT` | server | 64 | no |
|
||||
| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | server | 5 / 30 s | no |
|
||||
| `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | 1 / 64 / 8 | yes |
|
||||
|
||||
## Measurements
|
||||
|
||||
Measured on a development machine (uvicorn, heuristic AI, normal pacing:
|
||||
`AI_MOVE_DELAY` 0.8 s, one scripted human per game taking 1.5 s per move,
|
||||
4 players per game):
|
||||
|
||||
| Running games | Server CPU | Server memory | AI service CPU | AI memory |
|
||||
|---------------|-----------|---------------|----------------|-----------|
|
||||
| 0 | ~0 % | ~55–80 MB | ~0 % | ~30 MB |
|
||||
| 50 | 0.8 % of a core | ~88 MB | 0.5 % | ~30 MB |
|
||||
| 100 | 1.5 % of a core | ~99 MB | 0.9 % | ~30 MB |
|
||||
|
||||
Overload test: server limited to **15 % of one CPU**
|
||||
(`systemd-run --user --scope -p CPUQuota=15%`), 30 games playing at full speed
|
||||
(no delays, instant human moves):
|
||||
|
||||
| Time | Event |
|
||||
|------|-------|
|
||||
| 0 s | load starts |
|
||||
| ~1 s | `busy` |
|
||||
| ~3 s | `overloaded` (CPU at 100 % of quota, throttled 100 %) → `create_lobby` refused with `busy` |
|
||||
| … | running games keep playing (534 moves made while overloaded) |
|
||||
| ~14 s | games that finished could not restart → load gone → back to `normal` |
|
||||
|
||||
What this means:
|
||||
|
||||
- With realistic pacing a game costs almost nothing; **1 CPU is far more than
|
||||
50 games need**. `MAX_ACTIVE_GAMES=50` is a conservative safety cap, not a
|
||||
CPU requirement — raising it to a few hundred is fine with the heuristic AI.
|
||||
- Overload only happens with abnormal traffic (scripts, floods) or a heavy AI
|
||||
model. That is exactly what the limits are for.
|
||||
- A neural-network model will be the most expensive part. Measure its
|
||||
decision time and CPU with `python -m wizard_ai evaluate` and `docker stats`
|
||||
before raising the caps.
|
||||
|
||||
## Tuning guide
|
||||
|
||||
| Symptom | Adjust |
|
||||
|---------|--------|
|
||||
| "Server busy" although `/health` shows low `cpu` and `load: normal` | the game limit is reached: raise `MAX_ACTIVE_GAMES` (and `AI_MAX_INFLIGHT` to at least the same value) |
|
||||
| "Server busy" with high `cpu` / `throttled` | give the server more CPU (`SERVER_CPUS`) — or accept it: the server is protecting itself |
|
||||
| Friends behind one router get "server full" | raise `MAX_CONNECTIONS_PER_IP` |
|
||||
| Busy/overloaded flips too eagerly on a slow machine | raise `LOAD_LAG_*` thresholds or `LOAD_OVERLOAD_SAMPLES` |
|
||||
| AI plays weaker under load (logs: "Busy … using heuristic agent") | raise `AI_CPUS` and `AI_WORKERS` (thread-safe agent!) or use a smaller model; `AI_DEGRADE_QUEUE` sets how early it degrades |
|
||||
| Server logs "AI failed 5 times in a row, pausing AI requests" | the AI service hangs or crashes: check `docker compose logs wizard_ai` |
|
||||
| Server logs "… AI requests already waiting" | more simultaneous AI turns than `AI_MAX_INFLIGHT`: raise it to ≥ `MAX_ACTIVE_GAMES` |
|
||||
| Container restarts, `docker inspect` shows `OOMKilled: true` | raise `SERVER_MEMORY` / `AI_MEMORY` |
|
||||
|
||||
## Monitoring
|
||||
|
||||
- `curl -s https://game.marvinkrausser.com/wizard/api/health` — load, CPU,
|
||||
games, connections, AI state (see above).
|
||||
- `docker stats` — live CPU and memory per container against the caps.
|
||||
- Log lines to watch (`docker compose logs -f wizard_server wizard_ai`):
|
||||
|
||||
| Log line | Meaning |
|
||||
|----------|---------|
|
||||
| `Load level normal -> busy` / `busy -> overloaded` / `-> normal` | load level changes (with lag, CPU, throttling) |
|
||||
| `Refusing connection from <ip>: connection limit reached` | connection limits hit |
|
||||
| `Closing connection from <ip>: message flood` | a client was cut off |
|
||||
| `AI failed N times in a row, pausing AI requests for 30s` | circuit breaker opened |
|
||||
| `Playing random move for player N: …` | an AI move fell back to random (reason included) |
|
||||
| AI: `Overloaded: N requests in progress` / `Dropping expired request` / `Busy (N waiting): using heuristic agent` | the AI service shedding load |
|
||||
|
||||
## Testing it yourself
|
||||
|
||||
Automated tests:
|
||||
|
||||
- `wizard_server/tests/test_limits.py` — token bucket, limiters, load levels
|
||||
and hysteresis, cgroup parsing, AI circuit breaker and in-flight cap.
|
||||
- `wizard_server/tests/test_protection.py` — floods, connection limits, lobby
|
||||
creation rate, forced overload, `MAX_ACTIVE_GAMES`, a game continuing while
|
||||
overloaded, `server_status` messages.
|
||||
- `wizard_ai/tests/test_ai.py` — queue full, expired requests, degradation.
|
||||
|
||||
Manual checks:
|
||||
|
||||
```bash
|
||||
# Refusal UI without real load
|
||||
cd wizard_server && LOAD_FORCE_LEVEL=overloaded AI_SHARED_SECRET=dev uvicorn app.main:app
|
||||
|
||||
# Game limit: start one game, then a second lobby's "Start game" is refused
|
||||
cd wizard_server && MAX_ACTIVE_GAMES=1 AI_SHARED_SECRET=dev uvicorn app.main:app
|
||||
|
||||
# Real CPU exhaustion on Linux: cap the server at 15 % of a CPU,
|
||||
# then run many games without delays (e.g. a small websocket script)
|
||||
cd wizard_server && AI_MOVE_DELAY=0 TRICK_PAUSE=0 MAX_CONNECTIONS_PER_IP=200 \
|
||||
MESSAGE_RATE=1000 MESSAGE_BURST=1000 AI_SHARED_SECRET=dev \
|
||||
systemd-run --user --scope -p CPUQuota=15% uvicorn app.main:app
|
||||
```
|
||||
|
||||
## Known limitations
|
||||
|
||||
- A running game whose humans all disconnected keeps its `MAX_ACTIVE_GAMES`
|
||||
slot until their sessions expire (`SESSION_TTL`, 30 min), because they may
|
||||
reconnect. It costs no CPU meanwhile.
|
||||
- The limits are per server process. That is fine because the server must run
|
||||
as a single process anyway (all state is in memory).
|
||||
- Players behind one IP share `MAX_CONNECTIONS_PER_IP` and `LOBBY_CREATE_RATE`.
|
||||
- There is no waiting queue: players have to retry when the server is busy.
|
||||
+2
-1
@@ -11,11 +11,12 @@ to be read top to bottom the first time and used as a reference afterwards.
|
||||
| 1 | [Architecture overview](01-architecture.md) | The four components, how a move travels through the system, repository layout, key design decisions |
|
||||
| 2 | [Game rules as implemented](02-game-rules.md) | Every rule of Wizard as the code implements it, with the exact function that enforces it |
|
||||
| 3 | [Game engine (`wizard_game`)](03-game-engine.md) | The rules package: cards, the `WizardGame` state machine, observations, actions, events |
|
||||
| 4 | [Game server (`wizard_server/app`)](04-server.md) | Sessions, reconnects, lobbies, running games, AI turns, broadcasting, configuration |
|
||||
| 4 | [Game server (`wizard_server/app`)](04-server.md) | Sessions, reconnects, lobbies, running games, AI turns, broadcasting, limits (summary), configuration |
|
||||
| 5 | [Protocol reference](05-protocol.md) | Every websocket message between browser ↔ server and server ↔ AI, with examples |
|
||||
| 6 | [Frontend (`wizard_website`)](06-frontend.md) | The React app: connection hook, screens, components, styling, build |
|
||||
| 7 | [AI player (`wizard_ai`)](07-ai.md) | Agents, feature encoding, policy interface, training, evaluation, production service — and how to plug in a real model |
|
||||
| 8 | [Deployment, development & operations](08-operations.md) | Docker, networks, Caddy, environment variables, local setup, tests, troubleshooting, how-to recipes |
|
||||
| 9 | [Limits, load control and performance](09-limits-and-performance.md) | Resource caps, abuse limits, overload detection, what happens when compute runs out, AI cost bounds, measurements, tuning and monitoring |
|
||||
|
||||
## The system in one paragraph
|
||||
|
||||
|
||||
Reference in New Issue
Block a user