Wizard/docs/09-limits-and-performance.md

9. Limits, load control and performance

This chapter explains how Wizard keeps its resource use bounded, how it protects itself against abuse, and what happens when it runs out of compute. It covers all components, because the protection works in layers from Docker down to the browser.

Principles

1. Running games always continue. A game in progress is never stopped, slowed down on purpose or refused a move because of load.
2. New work is refused when there is no capacity. Creating lobbies and starting games is what adds load, so that is what gets refused — with a clear "server busy" message, not a timeout. (A waiting queue was considered and rejected as not worth the complexity.)
3. Abuse is rejected immediately. Floods and connection storms are cut off at the door, never queued.
4. The host is protected no matter what. Hard Docker caps guarantee that Wizard can never take CPU or memory from the other services on the machine, even if every software limit fails.
5. Degrade, don't fail. An overloaded or broken AI makes weaker moves (the heuristic, or a random move) instead of stopping games.

Overview

flowchart TB
    subgraph docker["Layer 1: Docker caps (compose.yaml)"]
        direction TB
        subgraph server["wizard_server — 1 CPU / 512 MB"]
            U["uvicorn: max message 64 KB,<br/>max 1000 connections"]
            A["Layer 2: abuse limits<br/>connections, message rate,<br/>lobby creation (limits.py)"]
            L["Layer 3: load monitor +<br/>admission control (load.py)"]
            B["AI bridge: in-flight cap,<br/>circuit breaker (ai_bridge.py)"]
        end
        subgraph ai["wizard_ai — 1 CPU / 1 GB"]
            Q["Layer 4: workers, queue,<br/>deadlines, degradation<br/>(service/client.py)"]
        end
        W["wizard_website — 0.25 CPU / 64 MB"]
    end
    Browser -->|websocket| U --> A --> L
    L -->|AI turns| B -->|decide| Q
    L -. "server_status" .-> Browser

Layer 1 — Docker caps

Set in compose.yaml from .env (defaults shown):

Container	CPU	Memory	Processes
wizard_server	SERVER_CPUS=1.0	SERVER_MEMORY=512M	256
wizard_ai	AI_CPUS=1.0	AI_MEMORY=1G	256
wizard_website	WEBSITE_CPUS=0.25	WEBSITE_MEMORY=64M	64

• CPU is a hard quota: when the container uses it up, the kernel throttles it. The server notices this (see layer 3) and refuses new games.
• Memory is a hard limit: a container exceeding it is killed by the kernel and restarted by Docker (restart: unless-stopped). For the server this would end all games (state is in memory), so the limit has lots of headroom: the server uses ~80 MB idle and ~100 MB with 100 games (see measurements).
• Processes (deploy.resources.limits.pids) stop fork bombs / thread explosions.
• The AI container sets OMP_NUM_THREADS / MKL_NUM_THREADS to AI_WORKERS, so numeric libraries (numpy, torch) don't start one thread per host core inside a 1-CPU container.

uvicorn (wizard_server/Dockerfile) adds transport limits: --ws-max-size 65536 (larger messages close the connection with code 1009; real messages are < 1 KB), --ws-max-queue 16, --limit-concurrency 1000 (HTTP 503 beyond that — a last resort above the app's own connection limit).

Layer 2 — abuse limits (server)

wizard_server/app/limits.py, applied in WizardService.handle_client (service.py):

Limit	Setting (default)	When exceeded
Open connections in total	MAX_CONNECTIONS (500)	error code busy, close 1013 "Try Again Later"
Open connections per client IP	MAX_CONNECTIONS_PER_IP (10)	same
Messages per connection	MESSAGE_RATE (10/s), MESSAGE_BURST (30)	message dropped; error code rate_limited (only on the 1st and every 10th drop)
Flooding	MAX_RATE_VIOLATIONS (50 drops)	connection closed with 1008
Lobbies created per client IP	LOBBY_CREATE_RATE (10 per minute)	error code rate_limited

Building blocks:

• TokenBucket(rate, burst) — the classic algorithm: a bucket holds up to burst tokens, refills at rate per second, each action takes one token. Allows short bursts (a player clicking quickly) but bounds the average.
• KeyedRateLimiter — one bucket per key (client IP); buckets that refilled completely are forgotten by the cleanup task every 30 s.
• ConnectionLimiter — counts open connections in total and per IP; released in the finally block of handle_client, so crashes cannot leak slots.

Client IP: websocket.client.host. Behind Caddy, uvicorn's --proxy-headers sets it from X-Forwarded-For; Caddy ignores client-supplied X-Forwarded-For values, so the IP cannot be spoofed. In local development everything comes from 127.0.0.1.

Normal play needs about one message per second, so these limits are never noticeable for humans. Several players behind one NAT (e.g. a household) share the per-IP connection limit — raise MAX_CONNECTIONS_PER_IP if that is a realistic use case.

Layer 3 — load monitor and admission control (server)

Detecting overload (load.py)

LoadMonitor.run() is started in the app lifespan and takes a sample every LOAD_SAMPLE_INTERVAL (1 s):

Signal	How it is measured	Why
Event loop lag	the monitor sleeps 1 s and measures how much later it woke up	the most direct measure of "the server reacts slowly"; works everywhere
CPU	process CPU time per second, relative to the container's CPU quota (at most 1 core, because game logic runs on one core)	detects saturation before players notice lag
Throttling	nr_throttled / nr_periods from cpu.stat	share of scheduling periods in which the container hit its Docker CPU cap

CPU quota and throttling come from the process's own cgroup (cgroup v2): /proc/self/cgroup gives its path, e.g. 0::/ inside a container (→ /sys/fs/cgroup) or 0::/user.slice/…scope on a host. Without cgroup files (e.g. macOS) the monitor still works with lag and CPU against one core.

Each sample is classified (classify), and next_level turns the recent classifications into the current level with hysteresis:

Level	Entered	Left
normal	—	—
busy	one sample with lag ≥ LOAD_LAG_BUSY (0.1 s) or CPU ≥ LOAD_CPU_BUSY (70 %)	after LOAD_RECOVER_SAMPLES (10) normal samples in a row
overloaded	LOAD_OVERLOAD_SAMPLES (3) samples in a row with lag ≥ LOAD_LAG_OVERLOAD (0.25 s), CPU ≥ LOAD_CPU_OVERLOAD (90 %) or throttled ≥ LOAD_THROTTLE_OVERLOAD (50 %) — or one sample with lag ≥ LOAD_LAG_SEVERE (1 s)	after 10 samples in a row that are not overloaded

Why hysteresis: without it a server at the edge would flip between states every second, and every flip is broadcast to all players. Why the "severe" rule: when the event loop is extremely late, samples themselves arrive late (one per several seconds), so waiting for three would delay detection a lot.

busy is informational only. LOAD_FORCE_LEVEL pins the level for testing.

Admission control (service.py)

WizardService.accepting_games is true unless the level is overloaded or MAX_ACTIVE_GAMES (50) games are running (LobbyManager.active_games()).

Action	While not accepting games
create_lobby	refused — error code busy
start_game	refused — error code busy
join_lobby	allowed (cheap)
moves in running games, return_to_lobby, leaving	allowed (protects running games)
reconnects (hello with token)	allowed

Refusing new games sheds load by itself: finished games are not replaced, so the load falls and the server recovers without intervention.

Telling the players

Every connected client gets a server_status message right after welcome and whenever the level or accepting_games changes:

{"type": "server_status", "load": "overloaded", "accepting_games": false}

Changes come from the load monitor (on_change) and from lobbies (on_status_change when a game starts, finishes or is abandoned), and are de-duplicated so an unchanged status is not re-sent.

Health endpoint

GET /health (public: https://game.marvinkrausser.com/wizard/api/health):

{
  "status": "ok",
  "load": "normal", "loop_lag": 0.0003, "cpu": 0.012, "throttled": 0.0,
  "accepting_games": true,
  "lobbies": 4, "active_games": 3, "sessions": 9, "connections": 7,
  "ai_available": true,
  "ai": {"connections": 1, "pending": 0, "paused": false}
}

status stays "ok" under load on purpose: the Docker health check must never mark an overloaded server unhealthy, because a restart would end every running game.

Layer 4 — AI cost bounds

In the server (ai_bridge.py)

Mechanism	Setting (default)	Behaviour
Deadline	AI_DECISION_TIMEOUT (5 s)	sent as timeout in every request; no answer in time → random move
In-flight cap	AI_MAX_INFLIGHT (64)	if that many requests already wait on an AI connection, the move is played randomly without asking. Keep it ≥ MAX_ACTIVE_GAMES: every game has at most one AI deciding at a time
Circuit breaker	AI_BREAKER_FAILURES (5), AI_BREAKER_COOLDOWN (30 s)	after 5 real failures in a row (timeouts, lost connection, malformed/illegal answers, unexpected errors) the AI is not asked for 30 s; games play random AI moves instead of waiting 5 s per move

What counts as a failure matters: the breaker is for a broken or hanging AI service. Deliberate load shedding — the in-flight cap, or the AI answering overloaded / expired — is not a failure: the service is healthy and answered immediately, and pausing it would only make play worse.

In the AI service (wizard_ai/wizard_ai/service/client.py)

flowchart TD
    R[decide request] --> Q{"in progress ≥<br/>AI_MAX_QUEUE?"}
    Q -- yes --> E1["error: overloaded"]
    Q -- no --> W[wait for a free worker<br/>AI_WORKERS]
    W --> X{"server timeout<br/>already passed?"}
    X -- yes --> E2["error: expired"]
    X -- no --> D{"≥ AI_DEGRADE_QUEUE others waiting,<br/>or primary agent slower than<br/>half the remaining time?"}
    D -- yes --> H[heuristic agent]
    D -- no --> M[primary agent / model]
    H --> OK["action + agent name"]
    M --> OK

Setting (default)	Meaning
AI_WORKERS (1)	decisions computed at the same time; agents must be thread safe if > 1
AI_MAX_QUEUE (64)	requests in progress (waiting + computing) before refusing with overloaded
AI_DEGRADE_QUEUE (8)	from this many other waiting requests, the heuristic decides instead of the model

The service also keeps a smoothed average of the primary agent's decision time; if it exceeds half of the time left before the server's deadline, the heuristic answers instead. The reply's agent field says who decided (the server logs it at debug level). With the heuristic as primary agent (today's default without a trained model) there is nothing cheaper to fall back to, so degradation only matters once a real model is deployed.

What players see

Situation	Start page	Lobby	Running game
Normal	—	—	—
Not accepting games (overloaded or game limit)	notice "The server is busy…", Create lobby disabled, Join works	notice, Start game shows "Server busy" (disabled)	unchanged
Connection limit reached	red banner "The server is full right now. Retrying in a moment…", retries after 15 s, 30 s, 60 s …	—	—
Message rate exceeded	toast "Too many messages, slow down."	same	same
AI overloaded / broken	—	—	AI players make weaker (heuristic or random) moves

Implemented in useWizard.ts (serverStatus, connection status busy), App.tsx, HomeScreen.tsx and LobbyScreen.tsx.

Configuration reference

All settings with their defaults. Server settings are read in wizard_server/app/config.py, AI settings in wizard_ai/wizard_ai/config.py. Settings marked compose are already wired through compose.yaml / .env; the others can be added to the environment: section of the service.

Variable	Component	Default	Wired in compose
SERVER_CPUS / SERVER_MEMORY	Docker	1.0 / 512M	yes
AI_CPUS / AI_MEMORY	Docker	1.0 / 1G	yes
WEBSITE_CPUS / WEBSITE_MEMORY	Docker	0.25 / 64M	yes
MAX_CONNECTIONS / MAX_CONNECTIONS_PER_IP	server	500 / 10	yes
MESSAGE_RATE / MESSAGE_BURST	server	10 / 30	no
MAX_RATE_VIOLATIONS	server	50	no
LOBBY_CREATE_RATE	server	10 per minute	no
MAX_LOBBIES	server	1000	no
MAX_ACTIVE_GAMES	server	50	yes
LOAD_SAMPLE_INTERVAL	server	1.0 s	no
LOAD_LAG_BUSY / LOAD_LAG_OVERLOAD / LOAD_LAG_SEVERE	server	0.1 / 0.25 / 1.0 s	no
LOAD_CPU_BUSY / LOAD_CPU_OVERLOAD	server	0.7 / 0.9	no
LOAD_THROTTLE_OVERLOAD	server	0.5	no
LOAD_OVERLOAD_SAMPLES / LOAD_RECOVER_SAMPLES	server	3 / 10	no
LOAD_FORCE_LEVEL	server	empty	no
AI_DECISION_TIMEOUT	server	5 s	no
AI_MAX_INFLIGHT	server	64	no
AI_BREAKER_FAILURES / AI_BREAKER_COOLDOWN	server	5 / 30 s	no
AI_WORKERS / AI_MAX_QUEUE / AI_DEGRADE_QUEUE	AI	1 / 64 / 8	yes

Measurements

Measured on a development machine (uvicorn, heuristic AI, normal pacing: AI_MOVE_DELAY 0.8 s, one scripted human per game taking 1.5 s per move, 4 players per game):

Running games	Server CPU	Server memory	AI service CPU	AI memory
0	~0 %	~55–80 MB	~0 %	~30 MB
50	0.8 % of a core	~88 MB	0.5 %	~30 MB
100	1.5 % of a core	~99 MB	0.9 %	~30 MB

Overload test: server limited to 15 % of one CPU (systemd-run --user --scope -p CPUQuota=15%), 30 games playing at full speed (no delays, instant human moves):

Time	Event
0 s	load starts
~1 s	busy
~3 s	overloaded (CPU at 100 % of quota, throttled 100 %) → create_lobby refused with busy
…	running games keep playing (534 moves made while overloaded)
~14 s	games that finished could not restart → load gone → back to normal

What this means:

• With realistic pacing a game costs almost nothing; 1 CPU is far more than 50 games need. MAX_ACTIVE_GAMES=50 is a conservative safety cap, not a CPU requirement — raising it to a few hundred is fine with the heuristic AI.
• Overload only happens with abnormal traffic (scripts, floods) or a heavy AI model. That is exactly what the limits are for.
• A neural-network model will be the most expensive part. Measure its decision time and CPU with python -m wizard_ai evaluate and docker stats before raising the caps.

Tuning guide

Symptom	Adjust
"Server busy" although /health shows low cpu and load: normal	the game limit is reached: raise MAX_ACTIVE_GAMES (and AI_MAX_INFLIGHT to at least the same value)
"Server busy" with high cpu / throttled	give the server more CPU (SERVER_CPUS) — or accept it: the server is protecting itself
Friends behind one router get "server full"	raise MAX_CONNECTIONS_PER_IP
Busy/overloaded flips too eagerly on a slow machine	raise LOAD_LAG_* thresholds or LOAD_OVERLOAD_SAMPLES
AI plays weaker under load (logs: "Busy … using heuristic agent")	raise AI_CPUS and AI_WORKERS (thread-safe agent!) or use a smaller model; AI_DEGRADE_QUEUE sets how early it degrades
Server logs "AI failed 5 times in a row, pausing AI requests"	the AI service hangs or crashes: check docker compose logs wizard_ai
Server logs "… AI requests already waiting"	more simultaneous AI turns than AI_MAX_INFLIGHT: raise it to ≥ MAX_ACTIVE_GAMES
Container restarts, docker inspect shows OOMKilled: true	raise SERVER_MEMORY / AI_MEMORY

Monitoring

• curl -s https://game.marvinkrausser.com/wizard/api/health — load, CPU, games, connections, AI state (see above).
• docker stats — live CPU and memory per container against the caps.
• Log lines to watch (docker compose logs -f wizard_server wizard_ai):

  Log line	Meaning
  Load level normal -> busy / busy -> overloaded / -> normal	load level changes (with lag, CPU, throttling)
  Refusing connection from <ip>: connection limit reached	connection limits hit
  Closing connection from <ip>: message flood	a client was cut off
  AI failed N times in a row, pausing AI requests for 30s	circuit breaker opened
  Playing random move for player N: …	an AI move fell back to random (reason included)
  AI: Overloaded: N requests in progress / Dropping expired request / Busy (N waiting): using heuristic agent	the AI service shedding load

Testing it yourself

Automated tests:

• wizard_server/tests/test_limits.py — token bucket, limiters, load levels and hysteresis, cgroup parsing, AI circuit breaker and in-flight cap.
• wizard_server/tests/test_protection.py — floods, connection limits, lobby creation rate, forced overload, MAX_ACTIVE_GAMES, a game continuing while overloaded, server_status messages.
• wizard_ai/tests/test_ai.py — queue full, expired requests, degradation.

Manual checks:

# Refusal UI without real load
cd wizard_server && LOAD_FORCE_LEVEL=overloaded AI_SHARED_SECRET=dev uvicorn app.main:app

# Game limit: start one game, then a second lobby's "Start game" is refused
cd wizard_server && MAX_ACTIVE_GAMES=1 AI_SHARED_SECRET=dev uvicorn app.main:app

# Real CPU exhaustion on Linux: cap the server at 15 % of a CPU,
# then run many games without delays (e.g. a small websocket script)
cd wizard_server && AI_MOVE_DELAY=0 TRICK_PAUSE=0 MAX_CONNECTIONS_PER_IP=200 \
  MESSAGE_RATE=1000 MESSAGE_BURST=1000 AI_SHARED_SECRET=dev \
  systemd-run --user --scope -p CPUQuota=15% uvicorn app.main:app

Known limitations

• A running game whose humans all disconnected keeps its MAX_ACTIVE_GAMES slot until their sessions expire (SESSION_TTL, 30 min), because they may reconnect. It costs no CPU meanwhile.
• The limits are per server process. That is fine because the server must run as a single process anyway (all state is in memory).
• Players behind one IP share MAX_CONNECTIONS_PER_IP and LOBBY_CREATE_RATE.
• There is no waiting queue: players have to retry when the server is busy.