Add resource limits, abuse protection and overload handling
Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.
- Docker: CPU/memory/pids caps per container (overridable in .env);
uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
rate per connection (token bucket, close 1008 on floods), lobby
creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
throttling, with hysteresis (normal/busy/overloaded, severe lag
overloads immediately)
- Admission control: create_lobby and start_game refused while
overloaded or MAX_ACTIVE_GAMES are running; server_status message
broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
limits; docs updated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+134
-13
@@ -26,7 +26,9 @@ flowchart TB
|
||||
| `service.py` | `WizardService` | the "controller": handles a browser connection, dispatches messages, manages sessions and cleanup |
|
||||
| `sessions.py` | `Session`, `Connection`, `SessionRegistry` | player identity across reconnects; ordered sending |
|
||||
| `lobby.py` | `LobbyManager`, `Lobby`, `Seat`, `LobbyError` | lobby membership, host actions, running the game, AI turns, broadcasting |
|
||||
| `ai_bridge.py` | `AIBridge`, `AIConnection` | talks to the AI service; random fallback |
|
||||
| `ai_bridge.py` | `AIBridge`, `AIConnection` | talks to the AI service; random fallback, in-flight cap, circuit breaker |
|
||||
| `limits.py` | `TokenBucket`, `KeyedRateLimiter`, `ConnectionLimiter` | abuse limits (message rate, connections, lobby creation) |
|
||||
| `load.py` | `LoadMonitor`, `next_level`, cgroup helpers | detects when the server runs out of compute |
|
||||
|
||||
## Routes (`main.py`)
|
||||
|
||||
@@ -249,7 +251,102 @@ The AI service keeps a websocket open to `/internal/ai`. `AIBridge`:
|
||||
the action is malformed or illegal (logged as warning, except the
|
||||
"no service connected" case which is logged at debug level);
|
||||
- calls `on_change` when a service connects/disconnects, and the service then
|
||||
re-broadcasts all lobbies so the lobby screen shows whether the AI is online.
|
||||
re-broadcasts all lobbies so the lobby screen shows whether the AI is online;
|
||||
- sends `timeout` with every request so the AI service can drop requests the
|
||||
server has already given up on;
|
||||
- **in-flight cap**: if `AI_MAX_INFLIGHT` requests are already waiting on a
|
||||
connection, the move is played by the fallback without asking;
|
||||
- **circuit breaker**: after `AI_BREAKER_FAILURES` failures in a row (timeouts,
|
||||
errors, illegal answers) the AI is not asked for `AI_BREAKER_COOLDOWN`
|
||||
seconds, so a struggling AI service is not buried under more requests.
|
||||
|
||||
## Limits and overload handling
|
||||
|
||||
The server protects the host and its players in three layers. The guiding rule
|
||||
is: **running games always continue; new work is refused when there is no
|
||||
capacity; abuse is rejected immediately.**
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
C[new websocket] --> L1{"connection limit<br/>total / per IP"}
|
||||
L1 -- over --> R1["error busy + close 1013"]
|
||||
L1 -- ok --> M[message]
|
||||
M --> L2{"message rate<br/>token bucket"}
|
||||
L2 -- over --> R2["dropped + error rate_limited<br/>(close 1008 when flooding)"]
|
||||
L2 -- ok --> D{"create_lobby /<br/>start_game?"}
|
||||
D -- no --> OK[handled normally]
|
||||
D -- yes --> L3{"accepting_games?<br/>not overloaded and<br/>running games < MAX_ACTIVE_GAMES"}
|
||||
L3 -- no --> R3["error busy"]
|
||||
L3 -- yes --> OK
|
||||
```
|
||||
|
||||
### 1. Hard limits (Docker and uvicorn)
|
||||
|
||||
- `compose.yaml` caps every container (`deploy.resources.limits` + `pids_limit`):
|
||||
server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB by default,
|
||||
overridable in `.env` (`SERVER_CPUS`, `SERVER_MEMORY`, …).
|
||||
- uvicorn (`wizard_server/Dockerfile`): `--ws-max-size 65536` (bigger messages
|
||||
close the connection with 1009), `--ws-max-queue 16`,
|
||||
`--limit-concurrency 1000` (HTTP 503 beyond that — a last resort above the
|
||||
app's own connection limit).
|
||||
|
||||
### 2. Abuse limits (`limits.py`, applied in `WizardService`)
|
||||
|
||||
| Limit | Default | When exceeded |
|
||||
|-------|---------|---------------|
|
||||
| open connections total / per client IP | 500 / 10 | `error` (code `busy`), then close **1013** "Try Again Later" |
|
||||
| messages per connection (token bucket) | 10/s, burst 30 | message dropped, `error` (code `rate_limited`, not repeated for every message); after 50 drops the connection is closed with 1008 |
|
||||
| lobbies created per client IP | 10 per minute | `error` (code `rate_limited`) |
|
||||
|
||||
The client IP is `websocket.client.host`. Behind Caddy, uvicorn's
|
||||
`--proxy-headers` sets it from `X-Forwarded-For` (Caddy does not pass on
|
||||
client-supplied values, so it cannot be spoofed).
|
||||
|
||||
### 3. Load monitor and admission control (`load.py`)
|
||||
|
||||
`LoadMonitor.run()` (started in the app lifespan) samples once per second:
|
||||
|
||||
- **event loop lag** — how late the loop wakes up; the most direct "players
|
||||
notice slowness" signal;
|
||||
- **CPU** — process CPU time relative to the container's CPU quota (read from
|
||||
the process's own cgroup, `/proc/self/cgroup` → `…/cpu.max`), at most one
|
||||
core because game logic runs on one core;
|
||||
- **throttling** — share of CPU periods in which the container hit its limit
|
||||
(`cpu.stat`: `nr_throttled / nr_periods`).
|
||||
|
||||
Each sample is classified (`classify`) and `next_level` derives the level with
|
||||
hysteresis:
|
||||
|
||||
| Level | Entered when | Left when |
|
||||
|-------|--------------|-----------|
|
||||
| `busy` | one sample with lag ≥ 0.1 s or CPU ≥ 70 % | 10 normal samples in a row |
|
||||
| `overloaded` | 3 samples in a row with lag ≥ 0.25 s, CPU ≥ 90 % or throttled ≥ 50 % — or **one** sample with lag ≥ 1 s | 10 samples in a row that are not overloaded |
|
||||
|
||||
`busy` is only informational. While **overloaded**, or while
|
||||
`MAX_ACTIVE_GAMES` (default 50) games are running, `accepting_games` is false:
|
||||
|
||||
| Action | Refused when not accepting games |
|
||||
|--------|----------------------------------|
|
||||
| `create_lobby` | yes (`error` code `busy`) |
|
||||
| `start_game` | yes (`error` code `busy`) |
|
||||
| `join_lobby`, moves, `return_to_lobby`, reconnects | never — they are cheap and keep running games alive |
|
||||
|
||||
Refusing new games sheds load by itself: finished games are not replaced, so
|
||||
the load drops and the server recovers.
|
||||
|
||||
Every connected client receives a `server_status` message after `welcome` and
|
||||
whenever the level or `accepting_games` changes (the website then disables
|
||||
"Create lobby" / "Start game" and shows a notice). `/health` reports `load`,
|
||||
`loop_lag`, `cpu`, `throttled`, `accepting_games`, `active_games`,
|
||||
`connections` and AI bridge stats, but its `status` **stays "ok"** under load:
|
||||
an overloaded server must never be restarted by Docker, which would end all
|
||||
running games.
|
||||
|
||||
Measured on a server capped at 15 % of a CPU with 30 games played at full
|
||||
speed: overloaded after ~3 s, new lobbies refused, running games kept going,
|
||||
back to normal ~10 s after the load dropped.
|
||||
|
||||
For manual testing, `LOAD_FORCE_LEVEL=overloaded` pins the level.
|
||||
|
||||
## Configuration (`config.py`)
|
||||
|
||||
@@ -265,27 +362,51 @@ All values are read from environment variables when the module is imported.
|
||||
| `MAX_LOBBIES` | 1000 | limit of simultaneously open lobbies |
|
||||
| `ALLOWED_ORIGINS` | empty (= all) | comma separated allowed `Origin` headers for `/ws` |
|
||||
| `LOG_LEVEL` | INFO | Python logging level |
|
||||
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | 500 / 10 | open websocket connections |
|
||||
| `MESSAGE_RATE` / `MESSAGE_BURST` | 10 / 30 | messages per second per connection |
|
||||
| `MAX_RATE_VIOLATIONS` | 50 | dropped messages before a flooding connection is closed |
|
||||
| `LOBBY_CREATE_RATE` | 10 | lobbies per IP per minute |
|
||||
| `MAX_ACTIVE_GAMES` | 50 | games running at the same time |
|
||||
| `LOAD_SAMPLE_INTERVAL` | 1.0 s | load sampling period |
|
||||
| `LOAD_LAG_BUSY` / `LOAD_LAG_OVERLOAD` / `LOAD_LAG_SEVERE` | 0.1 / 0.25 / 1.0 s | event loop lag thresholds |
|
||||
| `LOAD_CPU_BUSY` / `LOAD_CPU_OVERLOAD` | 0.7 / 0.9 | CPU share of the quota |
|
||||
| `LOAD_THROTTLE_OVERLOAD` | 0.5 | throttled share of CPU periods |
|
||||
| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | 3 / 10 | hysteresis |
|
||||
| `LOAD_FORCE_LEVEL` | empty | pin the level (testing) |
|
||||
| `AI_MAX_INFLIGHT` | 32 | waiting AI requests per AI connection |
|
||||
| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | 5 / 30 s | AI circuit breaker |
|
||||
|
||||
## Housekeeping
|
||||
|
||||
`WizardService.cleanup_loop()` runs every 30 s (started in the app lifespan)
|
||||
and removes expired sessions (leaving their lobby first).
|
||||
and removes expired sessions (leaving their lobby first) and forgets idle
|
||||
per-IP lobby-creation buckets.
|
||||
|
||||
## Tests (`wizard_server/tests/test_server.py`)
|
||||
## Tests (`wizard_server/tests/`)
|
||||
|
||||
Uses FastAPI's `TestClient` websockets with zero delays. Covers: health,
|
||||
hello requirement, validation errors, the full lobby flow including a complete
|
||||
20-round game between two humans and an AI, reconnect by token, leaving
|
||||
mid-game (seat becomes AI, host moves), the AI endpoint's secret and proxy
|
||||
checks, and a fake AI service answering decide requests.
|
||||
Uses FastAPI's `TestClient` websockets with zero delays and a raised message
|
||||
rate limit (`helpers.TEST_SETTINGS`).
|
||||
|
||||
Note for writing new tests: enter websocket sessions with
|
||||
`client.sockets.enter_context(client.websocket_connect(...))` (see the fixture),
|
||||
otherwise the test client hangs on shutdown.
|
||||
- `test_server.py` — health, hello requirement, validation errors, the full
|
||||
lobby flow including a complete 20-round game between two humans and an AI,
|
||||
reconnect by token, leaving mid-game (seat becomes AI, host moves), the AI
|
||||
endpoint's secret and proxy checks, and a fake AI service.
|
||||
- `test_protection.py` — message flood, connections per IP, lobby creation
|
||||
rate, forced overload, `MAX_ACTIVE_GAMES`, a running game continuing while
|
||||
overloaded, `server_status` messages.
|
||||
- `test_limits.py` — unit tests of the token bucket, limiters, load levels,
|
||||
cgroup parsing and the AI circuit breaker / in-flight cap.
|
||||
|
||||
Fixtures (`conftest.py`): `client` (default test settings) and
|
||||
`make_client(**overrides)` for custom settings. Enter websocket sessions with
|
||||
`client.sockets.enter_context(client.websocket_connect(...))` (as
|
||||
`helpers.connect` does), otherwise the test client hangs on shutdown.
|
||||
|
||||
## Limitations to be aware of
|
||||
|
||||
- In-memory state: restarting the server ends all lobbies and games.
|
||||
- Must run as a single process (no `--workers`).
|
||||
- No spectators: only seated players receive game state.
|
||||
- No rate limiting on websocket messages.
|
||||
- A running game whose humans all disconnected keeps its `MAX_ACTIVE_GAMES`
|
||||
slot until their sessions expire (`SESSION_TTL`), because they may still
|
||||
reconnect. It costs no CPU while waiting.
|
||||
|
||||
Reference in New Issue
Block a user