Wizard/docs/04-server.md

# 4. Game server (`wizard_server/app`)

A FastAPI application running in a single uvicorn process. It keeps
**everything in memory**: sessions, lobbies and running games.

## Module overview

```mermaid
flowchart TB
    main["main.py<br/>routes, auth"] --> service["service.py<br/>WizardService"]
    service --> sessions["sessions.py<br/>Session, Connection, SessionRegistry"]
    service --> lobby["lobby.py<br/>LobbyManager, Lobby, Seat"]
    service --> protocol["protocol.py<br/>message schema"]
    service --> bridge["ai_bridge.py<br/>AIBridge"]
    lobby --> bridge
    lobby --> engine[("wizard_game")]
    bridge --> engine
    main --> config["config.py<br/>Settings"]
```

| Module | Main types | Role |
|--------|-----------|------|
| `main.py` | `create_app()` | builds the FastAPI app, defines the three routes, starts the cleanup task |
| `config.py` | `Settings`, `settings` | configuration from environment variables |
| `protocol.py` | pydantic models, `client_message_adapter` | validates messages from browsers |
| `service.py` | `WizardService` | the "controller": handles a browser connection, dispatches messages, manages sessions and cleanup |
| `sessions.py` | `Session`, `Connection`, `SessionRegistry` | player identity across reconnects; ordered sending |
| `lobby.py` | `LobbyManager`, `Lobby`, `Seat`, `LobbyError` | lobby membership, host actions, running the game, AI turns, broadcasting |
| `ai_bridge.py` | `AIBridge`, `AIConnection` | talks to the AI service; random fallback, in-flight cap, circuit breaker |
| `limits.py` | `TokenBucket`, `KeyedRateLimiter`, `ConnectionLimiter` | abuse limits (message rate, connections, lobby creation) |
| `load.py` | `LoadMonitor`, `next_level`, cgroup helpers | detects when the server runs out of compute |

## Routes (`main.py`)

Caddy strips `/wizard/api`, so these are the paths the server sees:

| Route | Used by | Purpose |
|-------|---------|---------|
| `GET /health` | Docker health check, you | `{"status":"ok","lobbies":…,"sessions":…,"ai_available":…}` |
| `WS /ws` | browsers | the game protocol (see [protocol](05-protocol.md)) |
| `WS /internal/ai` | AI service | decision requests |

`/internal/ai` is protected twice:

1. `_via_proxy()` rejects the connection if it carries forwarding headers
   (`X-Forwarded-For`, `Forwarded`, `Via`). Caddy always adds them, so
   requests from the internet via `…/wizard/api/internal/ai` are refused. The
   AI service connects directly over the internal network and has none.
2. `_ai_authorized()` requires `Authorization: Bearer <AI_SHARED_SECRET>`
   (compared with `secrets.compare_digest`). If the secret is empty,
   authentication is disabled — only for development; a warning is logged at
   startup.

`create_app(config)` takes a `Settings` object so tests can build apps with
different configuration (e.g. zero delays).

## Core concepts

### Session — "who is this browser tab?"

`sessions.Session`:

| Field | Meaning |
|-------|---------|
| `player_id` | public id (12 hex chars), shown to other players, used as seat id |
| `token` | secret (`secrets.token_urlsafe(24)`), stored by the browser in `sessionStorage`, used to reconnect |
| `name` | display name, set when creating/joining a lobby |
| `lobby_code` | lobby the player is in, or `None` |
| `connection` | the current `Connection`, or `None` while disconnected |
| `last_seen` | monotonic time of the last disconnect/connect, for expiry |

`SessionRegistry` maps token → session. Sessions without a connection for
`SESSION_TTL` seconds (default 30 min) are removed by the cleanup task; if they
were in a lobby they leave it (see below).

### Connection — ordered sending

`sessions.Connection` wraps one websocket and owns an `asyncio.Queue`. A
background writer task sends queued messages one by one. Therefore:

- `session.send(msg)` is **synchronous** (just enqueues), so game logic never
  awaits while mutating state;
- messages to one client are always in order and never sent concurrently.

### Lobby and seats

`lobby.Seat` = one place at the table:

| Field | Meaning |
|-------|---------|
| `player_id` | session's player id, or `ai-xxxxxxxx` for AI seats |
| `name` | unique within the lobby (`"Ann"`, `"Ann (2)"`, …) |
| `is_ai` | AI seats are played via the `AIBridge` |
| `session` | the human's session (or `None`) |

The **seat index in `lobby.seats` is the player index in the game**. Seats are
never removed or reordered while a game runs, so indices stay valid.

`lobby.Lobby` fields: `code` (6 chars from `ABCDEFGHJKLMNPQRSTUVWXYZ23456789`, no
0/O/1/I), `seats`, `host_id`, `options` (`GameOptions`), `status`
(`waiting` → `playing` → `finished` → back to `waiting`), `game` (`WizardGame`
or `None`).

`LobbyManager` creates lobbies (up to `MAX_LOBBIES`), finds them by code
(case-insensitive) and deletes a lobby as soon as no human seat is left
(`discard_if_abandoned`).

## Lifecycle of a browser connection (`WizardService.handle_client`)

```mermaid
sequenceDiagram
    participant B as Browser
    participant S as WizardService
    B->>S: websocket connect /ws
    Note over S: Origin check (ALLOWED_ORIGINS), accept,<br/>start Connection writer
    B->>S: {"type":"hello","token": <stored token or null>}
    alt token known
        S->>S: reuse session (kick an older tab with close code 4000)
    else
        S->>S: create new session
    end
    S-->>B: {"type":"welcome","player_id","token","name"}
    S-->>B: {"type":"state", …} (lobby + game if still seated, else nulls)
    loop every message
        B->>S: create_lobby / join_lobby / action / …
        S->>S: validate (pydantic) → _dispatch → Lobby method
        S-->>B: state broadcast, or {"type":"error"}
    end
    B--xS: disconnect
    Note over S: session.connection = None, lobby broadcast<br/>(seat shows "offline"), the seat is kept
```

Rules applied in `_hello` and `_dispatch`:

- Every message except `hello` requires a completed `hello` on *this*
  connection.
- Invalid JSON/fields → `error` message with the pydantic reason; the
  connection stays open.
- `LobbyError` (rule violations like "Only the host can do that") → `error`
  message to that client only.
- Opening the same session in a second tab takes it over: the old tab gets an
  `error` and is closed with code **4000**; the frontend then stops
  reconnecting and shows "opened in another window".

### Message handling (`_dispatch`)

| Message | What happens |
|---------|-------------|
| `create_lobby {name}` | leave current lobby, set name, create lobby, join it (creator becomes host) |
| `join_lobby {code, name}` | lobby must exist; leave another lobby first; join. Joining is only possible while the lobby is `waiting` and has < 6 seats — **except** re-joining your own seat |
| `leave_lobby` | leave (see below), receive empty state |
| `add_ai` | host only, lobby waiting and not full; adds an AI seat named after a famous wizard (Merlin, Morgana, …) |
| `remove_player {player_id}` | host only, lobby waiting; the removed human gets an error + empty state |
| `set_options {hidden_bids}` | host only, lobby waiting |
| `start_game` | host only, 3–6 seats; creates `WizardGame(len(seats), options)` |
| `action {action}` | lobby must be `playing`; the engine validates the action |
| `return_to_lobby` | host only, game must be finished; status back to `waiting`, game discarded |

### Leaving

`Lobby.leave(session)`:

- In a **waiting** lobby the seat is removed.
- During a **playing or finished** game the seat is kept and **converted to an
  AI seat** (new `ai-…` id, same name), so the game continues.
- If the host leaves, the next human seat becomes host.
- If no human seat remains, the lobby is deleted (its AI task is cancelled).

Disconnecting (closing the tab, network loss) is *not* leaving: the seat stays
human and the game waits for that player. After `SESSION_TTL` without
reconnect, the cleanup task makes the session leave (→ AI takes over).

## Running a game

### Broadcasting state

`Lobby.broadcast(events)` sends every seated human their own message built by
`Lobby.state_for(session, events)`:

```json
{
  "type": "state",
  "lobby": { "code": "…", "status": "playing", "host_id": "…", "options": {…},
             "players": [{"id", "name", "is_ai", "connected"}, …],
             "min_players": 3, "max_players": 6, "ai_available": true },
  "game": { "you": 1, "observation": { … personal Observation … } },
  "events": [ … events caused by the last action … ]
}
```

It is called after every change (join, leave, options, start, every move, AI
service connecting/disconnecting, a player going offline).

### Human moves

`Lobby.apply_action(session, action_dict)` → `action_from_dict` →
`game.apply(seat, action)`. Errors become `LobbyError` → error toast in the
browser. On success `_after_action(events)` runs.

### `_after_action` and the AI loop

```python
def _after_action(self, events):
    if TrickCompleted in events: self._pause_until = now + TRICK_PAUSE
    if GameFinished in events:   self.status = FINISHED
    self.broadcast(events)
    self._schedule_ai()
```

`_schedule_ai()` starts `_ai_loop()` as an asyncio task if the current player
is an AI seat and no loop is running. The loop:

```mermaid
flowchart TD
    A{"current player<br/>is an AI seat?"} -- no --> Z[stop]
    A -- yes --> B["obs = game.observation(seat)"]
    B --> C["action = await bridge.decide(obs)"]
    C --> D["wait until max(start + AI_MOVE_DELAY, pause_until)"]
    D --> E{"same game, same seat,<br/>action still legal?"}
    E -- no --> A
    E -- yes --> F["game.apply → _after_action"]
    F --> A
```

- The AI "thinks" in parallel with the minimum delay, so a slow AI does not add
  to the delay.
- `TRICK_PAUSE` gives humans time to see a completed trick before AI players
  continue.
- The re-validation step makes the loop safe against state changes while it was
  awaiting (e.g. the host returned to the lobby).
- Several consecutive AI seats are handled by the same loop.

### Concurrency model

Everything runs on one asyncio event loop. State changes (join, apply action,
…) are **synchronous** — there is no `await` between checking and mutating —
so no locks are needed. The only awaits are in the AI loop (decide, sleep), and
it re-validates afterwards. This is also why the server must run with **one
worker process**: a second process would have its own, separate lobbies.

## AI bridge (`ai_bridge.py`)

The AI service keeps a websocket open to `/internal/ai`. `AIBridge`:

- keeps a list of connected AI services (`available` = at least one);
- `decide(observation)`: picks a connection round robin, sends
  `{"type":"decide","request_id","observation"}`, and awaits a future that
  `serve()` resolves when the matching reply arrives;
- falls back to a **random legal action** if no service is connected, the reply
  does not arrive within `AI_DECISION_TIMEOUT`, the service reports an error,
  the action is malformed or illegal (logged as warning, except the
  "no service connected" case which is logged at debug level);
- calls `on_change` when a service connects/disconnects, and the service then
  re-broadcasts all lobbies so the lobby screen shows whether the AI is online;
- sends `timeout` with every request so the AI service can drop requests the
  server has already given up on;
- **in-flight cap**: if `AI_MAX_INFLIGHT` requests are already waiting on a
  connection, the move is played by the fallback without asking;
- **circuit breaker**: after `AI_BREAKER_FAILURES` failures in a row (timeouts,
  errors, illegal answers) the AI is not asked for `AI_BREAKER_COOLDOWN`
  seconds, so a struggling AI service is not buried under more requests.

## Limits and overload handling

The server protects the host and its players in three layers. The guiding rule
is: **running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.**

```mermaid
flowchart TD
    C[new websocket] --> L1{"connection limit<br/>total / per IP"}
    L1 -- over --> R1["error busy + close 1013"]
    L1 -- ok --> M[message]
    M --> L2{"message rate<br/>token bucket"}
    L2 -- over --> R2["dropped + error rate_limited<br/>(close 1008 when flooding)"]
    L2 -- ok --> D{"create_lobby /<br/>start_game?"}
    D -- no --> OK[handled normally]
    D -- yes --> L3{"accepting_games?<br/>not overloaded and<br/>running games < MAX_ACTIVE_GAMES"}
    L3 -- no --> R3["error busy"]
    L3 -- yes --> OK
```

### 1. Hard limits (Docker and uvicorn)

- `compose.yaml` caps every container (`deploy.resources.limits` + `pids_limit`):
  server 1 CPU / 512 MB, AI 1 CPU / 1 GB, website 0.25 CPU / 64 MB by default,
  overridable in `.env` (`SERVER_CPUS`, `SERVER_MEMORY`, …).
- uvicorn (`wizard_server/Dockerfile`): `--ws-max-size 65536` (bigger messages
  close the connection with 1009), `--ws-max-queue 16`,
  `--limit-concurrency 1000` (HTTP 503 beyond that — a last resort above the
  app's own connection limit).

### 2. Abuse limits (`limits.py`, applied in `WizardService`)

| Limit | Default | When exceeded |
|-------|---------|---------------|
| open connections total / per client IP | 500 / 10 | `error` (code `busy`), then close **1013** "Try Again Later" |
| messages per connection (token bucket) | 10/s, burst 30 | message dropped, `error` (code `rate_limited`, not repeated for every message); after 50 drops the connection is closed with 1008 |
| lobbies created per client IP | 10 per minute | `error` (code `rate_limited`) |

The client IP is `websocket.client.host`. Behind Caddy, uvicorn's
`--proxy-headers` sets it from `X-Forwarded-For` (Caddy does not pass on
client-supplied values, so it cannot be spoofed).

### 3. Load monitor and admission control (`load.py`)

`LoadMonitor.run()` (started in the app lifespan) samples once per second:

- **event loop lag** — how late the loop wakes up; the most direct "players
  notice slowness" signal;
- **CPU** — process CPU time relative to the container's CPU quota (read from
  the process's own cgroup, `/proc/self/cgroup` → `…/cpu.max`), at most one
  core because game logic runs on one core;
- **throttling** — share of CPU periods in which the container hit its limit
  (`cpu.stat`: `nr_throttled / nr_periods`).

Each sample is classified (`classify`) and `next_level` derives the level with
hysteresis:

| Level | Entered when | Left when |
|-------|--------------|-----------|
| `busy` | one sample with lag ≥ 0.1 s or CPU ≥ 70 % | 10 normal samples in a row |
| `overloaded` | 3 samples in a row with lag ≥ 0.25 s, CPU ≥ 90 % or throttled ≥ 50 % — or **one** sample with lag ≥ 1 s | 10 samples in a row that are not overloaded |

`busy` is only informational. While **overloaded**, or while
`MAX_ACTIVE_GAMES` (default 50) games are running, `accepting_games` is false:

| Action | Refused when not accepting games |
|--------|----------------------------------|
| `create_lobby` | yes (`error` code `busy`) |
| `start_game` | yes (`error` code `busy`) |
| `join_lobby`, moves, `return_to_lobby`, reconnects | never — they are cheap and keep running games alive |

Refusing new games sheds load by itself: finished games are not replaced, so
the load drops and the server recovers.

Every connected client receives a `server_status` message after `welcome` and
whenever the level or `accepting_games` changes (the website then disables
"Create lobby" / "Start game" and shows a notice). `/health` reports `load`,
`loop_lag`, `cpu`, `throttled`, `accepting_games`, `active_games`,
`connections` and AI bridge stats, but its `status` **stays "ok"** under load:
an overloaded server must never be restarted by Docker, which would end all
running games.

Measured on a server capped at 15 % of a CPU with 30 games played at full
speed: overloaded after ~3 s, new lobbies refused, running games kept going,
back to normal ~10 s after the load dropped.

For manual testing, `LOAD_FORCE_LEVEL=overloaded` pins the level.

## Configuration (`config.py`)

All values are read from environment variables when the module is imported.

| Variable | Default | Meaning |
|----------|---------|---------|
| `AI_SHARED_SECRET` | empty | secret the AI service must send; empty = no auth (dev only) |
| `AI_DECISION_TIMEOUT` | 5.0 s | wait for the AI before playing a random move |
| `AI_MOVE_DELAY` | 0.8 s | minimum time an AI move takes, so humans can follow |
| `TRICK_PAUSE` | 1.8 s | pause after a completed trick before AIs continue |
| `SESSION_TTL` | 1800 s | disconnected sessions are dropped after this |
| `MAX_LOBBIES` | 1000 | limit of simultaneously open lobbies |
| `ALLOWED_ORIGINS` | empty (= all) | comma separated allowed `Origin` headers for `/ws` |
| `LOG_LEVEL` | INFO | Python logging level |
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | 500 / 10 | open websocket connections |
| `MESSAGE_RATE` / `MESSAGE_BURST` | 10 / 30 | messages per second per connection |
| `MAX_RATE_VIOLATIONS` | 50 | dropped messages before a flooding connection is closed |
| `LOBBY_CREATE_RATE` | 10 | lobbies per IP per minute |
| `MAX_ACTIVE_GAMES` | 50 | games running at the same time |
| `LOAD_SAMPLE_INTERVAL` | 1.0 s | load sampling period |
| `LOAD_LAG_BUSY` / `LOAD_LAG_OVERLOAD` / `LOAD_LAG_SEVERE` | 0.1 / 0.25 / 1.0 s | event loop lag thresholds |
| `LOAD_CPU_BUSY` / `LOAD_CPU_OVERLOAD` | 0.7 / 0.9 | CPU share of the quota |
| `LOAD_THROTTLE_OVERLOAD` | 0.5 | throttled share of CPU periods |
| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | 3 / 10 | hysteresis |
| `LOAD_FORCE_LEVEL` | empty | pin the level (testing) |
| `AI_MAX_INFLIGHT` | 32 | waiting AI requests per AI connection |
| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | 5 / 30 s | AI circuit breaker |

## Housekeeping

`WizardService.cleanup_loop()` runs every 30 s (started in the app lifespan)
and removes expired sessions (leaving their lobby first) and forgets idle
per-IP lobby-creation buckets.

## Tests (`wizard_server/tests/`)

Uses FastAPI's `TestClient` websockets with zero delays and a raised message
rate limit (`helpers.TEST_SETTINGS`).

- `test_server.py` — health, hello requirement, validation errors, the full
  lobby flow including a complete 20-round game between two humans and an AI,
  reconnect by token, leaving mid-game (seat becomes AI, host moves), the AI
  endpoint's secret and proxy checks, and a fake AI service.
- `test_protection.py` — message flood, connections per IP, lobby creation
  rate, forced overload, `MAX_ACTIVE_GAMES`, a running game continuing while
  overloaded, `server_status` messages.
- `test_limits.py` — unit tests of the token bucket, limiters, load levels,
  cgroup parsing and the AI circuit breaker / in-flight cap.

Fixtures (`conftest.py`): `client` (default test settings) and
`make_client(**overrides)` for custom settings. Enter websocket sessions with
`client.sockets.enter_context(client.websocket_connect(...))` (as
`helpers.connect` does), otherwise the test client hangs on shutdown.

## Limitations to be aware of

- In-memory state: restarting the server ends all lobbies and games.
- Must run as a single process (no `--workers`).
- No spectators: only seated players receive game state.
- A running game whose humans all disconnected keeps its `MAX_ACTIVE_GAMES`
  slot until their sessions expire (`SESSION_TTL`), because they may still
  reconnect. It costs no CPU while waiting.