Wizard/docs/04-server.md

4. Game server (wizard_server/app)

A FastAPI application running in a single uvicorn process. It keeps everything in memory: sessions, lobbies and running games.

Module overview

flowchart TB
    main["main.py<br/>routes, auth"] --> service["service.py<br/>WizardService"]
    service --> sessions["sessions.py<br/>Session, Connection, SessionRegistry"]
    service --> lobby["lobby.py<br/>LobbyManager, Lobby, Seat"]
    service --> protocol["protocol.py<br/>message schema"]
    service --> bridge["ai_bridge.py<br/>AIBridge"]
    lobby --> bridge
    lobby --> engine[("wizard_game")]
    bridge --> engine
    main --> config["config.py<br/>Settings"]

Module	Main types	Role
main.py	create_app()	builds the FastAPI app, defines the three routes, starts the cleanup task
config.py	Settings, settings	configuration from environment variables
protocol.py	pydantic models, client_message_adapter	validates messages from browsers
service.py	WizardService	the "controller": handles a browser connection, dispatches messages, manages sessions and cleanup
sessions.py	Session, Connection, SessionRegistry	player identity across reconnects; ordered sending
lobby.py	LobbyManager, Lobby, Seat, LobbyError	lobby membership, host actions, running the game, AI turns, broadcasting
ai_bridge.py	AIBridge, AIConnection	talks to the AI service; random fallback, in-flight cap, circuit breaker
limits.py	TokenBucket, KeyedRateLimiter, ConnectionLimiter	abuse limits (message rate, connections, lobby creation)
load.py	LoadMonitor, next_level, cgroup helpers	detects when the server runs out of compute

Routes (main.py)

Caddy strips /wizard/api, so these are the paths the server sees:

Route	Used by	Purpose
GET /health	Docker health check, you	{"status":"ok","lobbies":…,"sessions":…,"ai_available":…}
WS /ws	browsers	the game protocol (see protocol)
WS /internal/ai	AI service	decision requests

/internal/ai is protected twice:

1. _via_proxy() rejects the connection if it carries forwarding headers (X-Forwarded-For, Forwarded, Via). Caddy always adds them, so requests from the internet via …/wizard/api/internal/ai are refused. The AI service connects directly over the internal network and has none.
2. _ai_authorized() requires Authorization: Bearer <AI_SHARED_SECRET> (compared with secrets.compare_digest). If the secret is empty, authentication is disabled — only for development; a warning is logged at startup.

create_app(config) takes a Settings object so tests can build apps with different configuration (e.g. zero delays).

Core concepts

Session — "who is this browser tab?"

sessions.Session:

Field	Meaning
player_id	public id (12 hex chars), shown to other players, used as seat id
token	secret (secrets.token_urlsafe(24)), stored by the browser in sessionStorage, used to reconnect
name	display name, set when creating/joining a lobby
lobby_code	lobby the player is in, or None
connection	the current Connection, or None while disconnected
last_seen	monotonic time of the last disconnect/connect, for expiry

SessionRegistry maps token → session. Sessions without a connection for SESSION_TTL seconds (default 30 min) are removed by the cleanup task; if they were in a lobby they leave it (see below).

Connection — ordered sending

sessions.Connection wraps one websocket and owns an asyncio.Queue. A background writer task sends queued messages one by one. Therefore:

• session.send(msg) is synchronous (just enqueues), so game logic never awaits while mutating state;
• messages to one client are always in order and never sent concurrently.

Lobby and seats

lobby.Seat = one place at the table:

Field	Meaning
player_id	session's player id, or ai-xxxxxxxx for AI seats
name	unique within the lobby ("Ann", "Ann (2)", …)
is_ai	AI seats are played via the AIBridge
session	the human's session (or None)

The seat index in lobby.seats is the player index in the game. Seats are never removed or reordered while a game runs, so indices stay valid.

lobby.Lobby fields: code (6 chars from ABCDEFGHJKLMNPQRSTUVWXYZ23456789, no 0/O/1/I), seats, host_id, options (GameOptions), status (waiting → playing → finished → back to waiting), game (WizardGame or None).

LobbyManager creates lobbies (up to MAX_LOBBIES), finds them by code (case-insensitive) and deletes a lobby as soon as no human seat is left (discard_if_abandoned).

Lifecycle of a browser connection (WizardService.handle_client)

sequenceDiagram
    participant B as Browser
    participant S as WizardService
    B->>S: websocket connect /ws
    Note over S: Origin check (ALLOWED_ORIGINS), accept,<br/>start Connection writer
    B->>S: {"type":"hello","token": <stored token or null>}
    alt token known
        S->>S: reuse session (kick an older tab with close code 4000)
    else
        S->>S: create new session
    end
    S-->>B: {"type":"welcome","player_id","token","name"}
    S-->>B: {"type":"state", …} (lobby + game if still seated, else nulls)
    loop every message
        B->>S: create_lobby / join_lobby / action / …
        S->>S: validate (pydantic) → _dispatch → Lobby method
        S-->>B: state broadcast, or {"type":"error"}
    end
    B--xS: disconnect
    Note over S: session.connection = None, lobby broadcast<br/>(seat shows "offline"), the seat is kept

Rules applied in _hello and _dispatch:

• Every message except hello requires a completed hello on this connection.
• Invalid JSON/fields → error message with the pydantic reason; the connection stays open.
• LobbyError (rule violations like "Only the host can do that") → error message to that client only.
• Opening the same session in a second tab takes it over: the old tab gets an error and is closed with code 4000; the frontend then stops reconnecting and shows "opened in another window".

Message handling (_dispatch)

Message	What happens
create_lobby {name}	leave current lobby, set name, create lobby, join it (creator becomes host)
join_lobby {code, name}	lobby must exist; leave another lobby first; join. Joining is only possible while the lobby is waiting and has < 6 seats — except re-joining your own seat
leave_lobby	leave (see below), receive empty state
add_ai	host only, lobby waiting and not full; adds an AI seat named after a famous wizard (Merlin, Morgana, …)
remove_player {player_id}	host only, lobby waiting; the removed human gets an error + empty state
set_options {hidden_bids}	host only, lobby waiting
start_game	host only, 3–6 seats; creates WizardGame(len(seats), options)
action {action}	lobby must be playing; the engine validates the action
return_to_lobby	host only, game must be finished; status back to waiting, game discarded

Leaving

Lobby.leave(session):

• In a waiting lobby the seat is removed.
• During a playing or finished game the seat is kept and converted to an AI seat (new ai-… id, same name), so the game continues.
• If the host leaves, the next human seat becomes host.
• If no human seat remains, the lobby is deleted (its AI task is cancelled).

Disconnecting (closing the tab, network loss) is not leaving: the seat stays human and the game waits for that player. After SESSION_TTL without reconnect, the cleanup task makes the session leave (→ AI takes over).

Running a game

Broadcasting state

Lobby.broadcast(events) sends every seated human their own message built by Lobby.state_for(session, events):

{
  "type": "state",
  "lobby": { "code": "…", "status": "playing", "host_id": "…", "options": {…},
             "players": [{"id", "name", "is_ai", "connected"}, …],
             "min_players": 3, "max_players": 6, "ai_available": true },
  "game": { "you": 1, "observation": { … personal Observation … } },
  "events": [ … events caused by the last action … ]
}

It is called after every change (join, leave, options, start, every move, AI service connecting/disconnecting, a player going offline).

Human moves

Lobby.apply_action(session, action_dict) → action_from_dict → game.apply(seat, action). Errors become LobbyError → error toast in the browser. On success _after_action(events) runs.

_after_action and the AI loop

def _after_action(self, events):
    if TrickCompleted in events: self._pause_until = now + TRICK_PAUSE
    if GameFinished in events:   self.status = FINISHED
    self.broadcast(events)
    self._schedule_ai()

_schedule_ai() starts _ai_loop() as an asyncio task if the current player is an AI seat and no loop is running. The loop:

flowchart TD
    A{"current player<br/>is an AI seat?"} -- no --> Z[stop]
    A -- yes --> B["obs = game.observation(seat)"]
    B --> C["action = await bridge.decide(obs)"]
    C --> D["wait until max(start + AI_MOVE_DELAY, pause_until)"]
    D --> E{"same game, same seat,<br/>action still legal?"}
    E -- no --> A
    E -- yes --> F["game.apply → _after_action"]
    F --> A

• The AI "thinks" in parallel with the minimum delay, so a slow AI does not add to the delay.
• TRICK_PAUSE gives humans time to see a completed trick before AI players continue.
• The re-validation step makes the loop safe against state changes while it was awaiting (e.g. the host returned to the lobby).
• Several consecutive AI seats are handled by the same loop.

Concurrency model

Everything runs on one asyncio event loop. State changes (join, apply action, …) are synchronous — there is no await between checking and mutating — so no locks are needed. The only awaits are in the AI loop (decide, sleep), and it re-validates afterwards. This is also why the server must run with one worker process: a second process would have its own, separate lobbies.

AI bridge (ai_bridge.py)

The AI service keeps a websocket open to /internal/ai. AIBridge:

• keeps a list of connected AI services (available = at least one);
• decide(observation): picks a connection round robin, sends {"type":"decide","request_id","observation"}, and awaits a future that serve() resolves when the matching reply arrives;
• falls back to a random legal action if no service is connected, the reply does not arrive within AI_DECISION_TIMEOUT, the service reports an error, the action is malformed or illegal (logged as warning, except the "no service connected" case which is logged at debug level);
• calls on_change when a service connects/disconnects, and the service then re-broadcasts all lobbies so the lobby screen shows whether the AI is online;
• sends timeout with every request so the AI service can drop requests the server has already given up on;
• in-flight cap: if AI_MAX_INFLIGHT requests are already waiting on a connection, the move is played by the fallback without asking;
• circuit breaker: after AI_BREAKER_FAILURES failures in a row (timeouts, errors, illegal answers) the AI is not asked for AI_BREAKER_COOLDOWN seconds, so a struggling AI service is not buried under more requests.

Limits and overload handling

The server protects the host and its players in three layers. The guiding rule is: running games always continue; new work is refused when there is no capacity; abuse is rejected immediately.

flowchart TD
    C[new websocket] --> L1{"connection limit<br/>total / per IP"}
    L1 -- over --> R1["error busy + close 1013"]
    L1 -- ok --> M[message]
    M --> L2{"message rate<br/>token bucket"}
    L2 -- over --> R2["dropped + error rate_limited<br/>(close 1008 when flooding)"]
    L2 -- ok --> D{"create_lobby /<br/>start_game?"}
    D -- no --> OK[handled normally]
    D -- yes --> L3{"accepting_games?<br/>not overloaded and<br/>running games < MAX_ACTIVE_GAMES"}
    L3 -- no --> R3["error busy"]
    L3 -- yes --> OK

1. Hard limits (Docker and uvicorn)

• compose.yaml caps every container (deploy.resources.limits + pids_limit): server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB by default, overridable in .env (SERVER_CPUS, SERVER_MEMORY, …).
• uvicorn (wizard_server/Dockerfile): --ws-max-size 65536 (bigger messages close the connection with 1009), --ws-max-queue 16, --limit-concurrency 1000 (HTTP 503 beyond that — a last resort above the app's own connection limit).

2. Abuse limits (limits.py, applied in WizardService)

Limit	Default	When exceeded
open connections total / per client IP	500 / 10	error (code busy), then close 1013 "Try Again Later"
messages per connection (token bucket)	10/s, burst 30	message dropped, error (code rate_limited, not repeated for every message); after 50 drops the connection is closed with 1008
lobbies created per client IP	10 per minute	error (code rate_limited)

The client IP is websocket.client.host. Behind Caddy, uvicorn's --proxy-headers sets it from X-Forwarded-For (Caddy does not pass on client-supplied values, so it cannot be spoofed).

3. Load monitor and admission control (load.py)

LoadMonitor.run() (started in the app lifespan) samples once per second:

• event loop lag — how late the loop wakes up; the most direct "players notice slowness" signal;
• CPU — process CPU time relative to the container's CPU quota (read from the process's own cgroup, /proc/self/cgroup → …/cpu.max), at most one core because game logic runs on one core;
• throttling — share of CPU periods in which the container hit its limit (cpu.stat: nr_throttled / nr_periods).

Each sample is classified (classify) and next_level derives the level with hysteresis:

Level	Entered when	Left when
busy	one sample with lag ≥ 0.1 s or CPU ≥ 70 %	10 normal samples in a row
overloaded	3 samples in a row with lag ≥ 0.25 s, CPU ≥ 90 % or throttled ≥ 50 % — or one sample with lag ≥ 1 s	10 samples in a row that are not overloaded

busy is only informational. While overloaded, or while MAX_ACTIVE_GAMES (default 50) games are running, accepting_games is false:

Action	Refused when not accepting games
create_lobby	yes (error code busy)
start_game	yes (error code busy)
join_lobby, moves, return_to_lobby, reconnects	never — they are cheap and keep running games alive

Refusing new games sheds load by itself: finished games are not replaced, so the load drops and the server recovers.

Every connected client receives a server_status message after welcome and whenever the level or accepting_games changes (the website then disables "Create lobby" / "Start game" and shows a notice). /health reports load, loop_lag, cpu, throttled, accepting_games, active_games, connections and AI bridge stats, but its status stays "ok" under load: an overloaded server must never be restarted by Docker, which would end all running games.

Measured on a server capped at 15 % of a CPU with 30 games played at full speed: overloaded after ~3 s, new lobbies refused, running games kept going, back to normal ~10 s after the load dropped.

For manual testing, LOAD_FORCE_LEVEL=overloaded pins the level.

Configuration (config.py)

All values are read from environment variables when the module is imported.

Variable	Default	Meaning
AI_SHARED_SECRET	empty	secret the AI service must send; empty = no auth (dev only)
AI_DECISION_TIMEOUT	5.0 s	wait for the AI before playing a random move
AI_MOVE_DELAY	0.8 s	minimum time an AI move takes, so humans can follow
TRICK_PAUSE	1.8 s	pause after a completed trick before AIs continue
SESSION_TTL	1800 s	disconnected sessions are dropped after this
MAX_LOBBIES	1000	limit of simultaneously open lobbies
ALLOWED_ORIGINS	empty (= all)	comma separated allowed Origin headers for /ws
LOG_LEVEL	INFO	Python logging level
MAX_CONNECTIONS / MAX_CONNECTIONS_PER_IP	500 / 10	open websocket connections
MESSAGE_RATE / MESSAGE_BURST	10 / 30	messages per second per connection
MAX_RATE_VIOLATIONS	50	dropped messages before a flooding connection is closed
LOBBY_CREATE_RATE	10	lobbies per IP per minute
MAX_ACTIVE_GAMES	50	games running at the same time
LOAD_SAMPLE_INTERVAL	1.0 s	load sampling period
LOAD_LAG_BUSY / LOAD_LAG_OVERLOAD / LOAD_LAG_SEVERE	0.1 / 0.25 / 1.0 s	event loop lag thresholds
LOAD_CPU_BUSY / LOAD_CPU_OVERLOAD	0.7 / 0.9	CPU share of the quota
LOAD_THROTTLE_OVERLOAD	0.5	throttled share of CPU periods
LOAD_OVERLOAD_SAMPLES / LOAD_RECOVER_SAMPLES	3 / 10	hysteresis
LOAD_FORCE_LEVEL	empty	pin the level (testing)
AI_MAX_INFLIGHT	32	waiting AI requests per AI connection
AI_BREAKER_FAILURES / AI_BREAKER_COOLDOWN	5 / 30 s	AI circuit breaker

Housekeeping

WizardService.cleanup_loop() runs every 30 s (started in the app lifespan) and removes expired sessions (leaving their lobby first) and forgets idle per-IP lobby-creation buckets.

Tests (wizard_server/tests/)

Uses FastAPI's TestClient websockets with zero delays and a raised message rate limit (helpers.TEST_SETTINGS).

• test_server.py — health, hello requirement, validation errors, the full lobby flow including a complete 20-round game between two humans and an AI, reconnect by token, leaving mid-game (seat becomes AI, host moves), the AI endpoint's secret and proxy checks, and a fake AI service.
• test_protection.py — message flood, connections per IP, lobby creation rate, forced overload, MAX_ACTIVE_GAMES, a running game continuing while overloaded, server_status messages.
• test_limits.py — unit tests of the token bucket, limiters, load levels, cgroup parsing and the AI circuit breaker / in-flight cap.

Fixtures (conftest.py): client (default test settings) and make_client(**overrides) for custom settings. Enter websocket sessions with client.sockets.enter_context(client.websocket_connect(...)) (as helpers.connect does), otherwise the test client hangs on shutdown.

Limitations to be aware of

• In-memory state: restarting the server ends all lobbies and games.
• Must run as a single process (no --workers).
• No spectators: only seated players receive game state.
• A running game whose humans all disconnected keeps its MAX_ACTIVE_GAMES slot until their sessions expire (SESSION_TTL), because they may still reconnect. It costs no CPU while waiting.