Wizard/docs/04-server.md

# 4. Game server (`wizard_server/app`)

A FastAPI application running in a single uvicorn process. It keeps
**everything in memory**: sessions, lobbies and running games.

## Module overview

```mermaid
flowchart TB
    main["main.py<br/>routes, auth"] --> service["service.py<br/>WizardService"]
    service --> sessions["sessions.py<br/>Session, Connection, SessionRegistry"]
    service --> lobby["lobby.py<br/>LobbyManager, Lobby, Seat"]
    service --> protocol["protocol.py<br/>message schema"]
    service --> bridge["ai_bridge.py<br/>AIBridge"]
    lobby --> bridge
    lobby --> engine[("wizard_game")]
    bridge --> engine
    main --> config["config.py<br/>Settings"]
```

| Module | Main types | Role |
|--------|-----------|------|
| `main.py` | `create_app()` | builds the FastAPI app, defines the three routes, starts the cleanup task |
| `config.py` | `Settings`, `settings` | configuration from environment variables |
| `protocol.py` | pydantic models, `client_message_adapter` | validates messages from browsers |
| `service.py` | `WizardService` | the "controller": handles a browser connection, dispatches messages, manages sessions and cleanup |
| `sessions.py` | `Session`, `Connection`, `SessionRegistry` | player identity across reconnects; ordered sending |
| `lobby.py` | `LobbyManager`, `Lobby`, `Seat`, `LobbyError` | lobby membership, host actions, running the game, AI turns, broadcasting |
| `ai_bridge.py` | `AIBridge`, `AIConnection` | talks to the AI service; random fallback, in-flight cap, circuit breaker |
| `limits.py` | `TokenBucket`, `KeyedRateLimiter`, `ConnectionLimiter` | abuse limits (message rate, connections, lobby creation) |
| `load.py` | `LoadMonitor`, `next_level`, cgroup helpers | detects when the server runs out of compute |

## Routes (`main.py`)

Caddy strips `/wizard/api`, so these are the paths the server sees:

| Route | Used by | Purpose |
|-------|---------|---------|
| `GET /health` | Docker health check, you | `{"status":"ok","lobbies":…,"sessions":…,"ai_available":…}` |
| `WS /ws` | browsers | the game protocol (see [protocol](05-protocol.md)) |
| `WS /internal/ai` | AI service | decision requests |

`/internal/ai` is protected twice:

1. `_via_proxy()` rejects the connection if it carries forwarding headers
   (`X-Forwarded-For`, `Forwarded`, `Via`). Caddy always adds them, so
   requests from the internet via `…/wizard/api/internal/ai` are refused. The
   AI service connects directly over the internal network and has none.
2. `_ai_authorized()` requires `Authorization: Bearer <AI_SHARED_SECRET>`
   (compared with `secrets.compare_digest`). If the secret is empty,
   authentication is disabled — only for development; a warning is logged at
   startup.

`create_app(config)` takes a `Settings` object so tests can build apps with
different configuration (e.g. zero delays).

## Core concepts

### Session — "who is this browser tab?"

`sessions.Session`:

| Field | Meaning |
|-------|---------|
| `player_id` | public id (12 hex chars), shown to other players, used as seat id |
| `token` | secret (`secrets.token_urlsafe(24)`), stored by the browser in `sessionStorage`, used to reconnect |
| `name` | display name, set when creating/joining a lobby |
| `lobby_code` | lobby the player is in, or `None` |
| `connection` | the current `Connection`, or `None` while disconnected |
| `last_seen` | monotonic time of the last disconnect/connect, for expiry |

`SessionRegistry` maps token → session. Sessions without a connection for
`SESSION_TTL` seconds (default 30 min) are removed by the cleanup task; if they
were in a lobby they leave it (see below).

### Connection — ordered sending

`sessions.Connection` wraps one websocket and owns an `asyncio.Queue`. A
background writer task sends queued messages one by one. Therefore:

- `session.send(msg)` is **synchronous** (just enqueues), so game logic never
  awaits while mutating state;
- messages to one client are always in order and never sent concurrently.

### Lobby and seats

`lobby.Seat` = one place at the table:

| Field | Meaning |
|-------|---------|
| `player_id` | session's player id, or `ai-xxxxxxxx` for AI seats |
| `name` | unique within the lobby (`"Ann"`, `"Ann (2)"`, …) |
| `is_ai` | AI seats are played via the `AIBridge` |
| `session` | the human's session (or `None`) |

The **seat index in `lobby.seats` is the player index in the game**. Seats are
never removed or reordered while a game runs, so indices stay valid.

`lobby.Lobby` fields: `code` (6 chars from `ABCDEFGHJKLMNPQRSTUVWXYZ23456789`, no
0/O/1/I), `seats`, `host_id`, `options` (`GameOptions`), `status`
(`waiting` → `playing` → `finished` → back to `waiting`), `game` (`WizardGame`
or `None`).

`LobbyManager` creates lobbies (up to `MAX_LOBBIES`), finds them by code
(case-insensitive) and deletes a lobby as soon as no human seat is left
(`discard_if_abandoned`).

## Lifecycle of a browser connection (`WizardService.handle_client`)

```mermaid
sequenceDiagram
    participant B as Browser
    participant S as WizardService
    B->>S: websocket connect /ws
    Note over S: Origin check (ALLOWED_ORIGINS), accept,<br/>start Connection writer
    B->>S: {"type":"hello","token": <stored token or null>}
    alt token known
        S->>S: reuse session (kick an older tab with close code 4000)
    else
        S->>S: create new session
    end
    S-->>B: {"type":"welcome","player_id","token","name"}
    S-->>B: {"type":"state", …} (lobby + game if still seated, else nulls)
    loop every message
        B->>S: create_lobby / join_lobby / action / …
        S->>S: validate (pydantic) → _dispatch → Lobby method
        S-->>B: state broadcast, or {"type":"error"}
    end
    B--xS: disconnect
    Note over S: session.connection = None, lobby broadcast<br/>(seat shows "offline"), the seat is kept
```

Rules applied in `_hello` and `_dispatch`:

- Every message except `hello` requires a completed `hello` on *this*
  connection.
- Invalid JSON/fields → `error` message with the pydantic reason; the
  connection stays open.
- `LobbyError` (rule violations like "Only the host can do that") → `error`
  message to that client only.
- Opening the same session in a second tab takes it over: the old tab gets an
  `error` and is closed with code **4000**; the frontend then stops
  reconnecting and shows "opened in another window".

### Message handling (`_dispatch`)

| Message | What happens |
|---------|-------------|
| `create_lobby {name}` | leave current lobby, set name, create lobby, join it (creator becomes host) |
| `join_lobby {code, name}` | lobby must exist; leave another lobby first; join. Joining is only possible while the lobby is `waiting` and has < 6 seats — **except** re-joining your own seat |
| `leave_lobby` | leave (see below), receive empty state |
| `add_ai` | host only, lobby waiting and not full; adds an AI seat named after a famous wizard (Merlin, Morgana, …) |
| `remove_player {player_id}` | host only, lobby waiting; the removed human gets an error + empty state |
| `set_options {hidden_bids}` | host only, lobby waiting |
| `start_game` | host only, 3–6 seats; creates `WizardGame(len(seats), options)` |
| `action {action}` | lobby must be `playing`; the engine validates the action |
| `return_to_lobby` | host only, game must be finished; status back to `waiting`, game discarded |

### Leaving

`Lobby.leave(session)`:

- In a **waiting** lobby the seat is removed.
- During a **playing or finished** game the seat is kept and **converted to an
  AI seat** (new `ai-…` id, same name), so the game continues.
- If the host leaves, the next human seat becomes host.
- If no human seat remains, the lobby is deleted (its AI task is cancelled).

Disconnecting (closing the tab, network loss) is *not* leaving: the seat stays
human and the game waits for that player. After `SESSION_TTL` without
reconnect, the cleanup task makes the session leave (→ AI takes over).

## Running a game

### Broadcasting state

`Lobby.broadcast(events)` sends every seated human their own message built by
`Lobby.state_for(session, events)`:

```json
{
  "type": "state",
  "lobby": { "code": "…", "status": "playing", "host_id": "…", "options": {…},
             "players": [{"id", "name", "is_ai", "connected"}, …],
             "min_players": 3, "max_players": 6, "ai_available": true },
  "game": { "you": 1, "observation": { … personal Observation … } },
  "events": [ … events caused by the last action … ]
}
```

It is called after every change (join, leave, options, start, every move, AI
service connecting/disconnecting, a player going offline).

### Human moves

`Lobby.apply_action(session, action_dict)` → `action_from_dict` →
`game.apply(seat, action)`. Errors become `LobbyError` → error toast in the
browser. On success `_after_action(events)` runs.

### `_after_action` and the AI loop

```python
def _after_action(self, events):
    if TrickCompleted in events: self._pause_until = now + TRICK_PAUSE
    if GameFinished in events:   self.status = FINISHED
    self.broadcast(events)
    self._schedule_ai()
```

`_schedule_ai()` starts `_ai_loop()` as an asyncio task if the current player
is an AI seat and no loop is running. The loop:

```mermaid
flowchart TD
    A{"current player<br/>is an AI seat?"} -- no --> Z[stop]
    A -- yes --> B["obs = game.observation(seat)"]
    B --> C["action = await bridge.decide(obs)"]
    C --> D["wait until max(start + AI_MOVE_DELAY, pause_until)"]
    D --> E{"same game, same seat,<br/>action still legal?"}
    E -- no --> A
    E -- yes --> F["game.apply → _after_action"]
    F --> A
```

- The AI "thinks" in parallel with the minimum delay, so a slow AI does not add
  to the delay.
- `TRICK_PAUSE` gives humans time to see a completed trick before AI players
  continue.
- The re-validation step makes the loop safe against state changes while it was
  awaiting (e.g. the host returned to the lobby).
- Several consecutive AI seats are handled by the same loop.

### Concurrency model

Everything runs on one asyncio event loop. State changes (join, apply action,
…) are **synchronous** — there is no `await` between checking and mutating —
so no locks are needed. The only awaits are in the AI loop (decide, sleep), and
it re-validates afterwards. This is also why the server must run with **one
worker process**: a second process would have its own, separate lobbies.

## AI bridge (`ai_bridge.py`)

The AI service keeps a websocket open to `/internal/ai`. `AIBridge`:

- keeps a list of connected AI services (`available` = at least one);
- `decide(observation)`: picks a connection round robin, sends
  `{"type":"decide","request_id","observation"}`, and awaits a future that
  `serve()` resolves when the matching reply arrives;
- falls back to a **random legal action** if no service is connected, the reply
  does not arrive within `AI_DECISION_TIMEOUT`, the service reports an error,
  the action is malformed or illegal (logged as warning, except the
  "no service connected" case which is logged at debug level);
- calls `on_change` when a service connects/disconnects, and the service then
  re-broadcasts all lobbies so the lobby screen shows whether the AI is online;
- sends `timeout` with every request so the AI service can drop requests the
  server has already given up on;
- **in-flight cap**: if `AI_MAX_INFLIGHT` requests are already waiting on a
  connection, the move is played by the fallback without asking;
- **circuit breaker**: after `AI_BREAKER_FAILURES` real failures in a row
  (timeouts, lost connection, malformed/illegal answers, unexpected errors) the
  AI is not asked for `AI_BREAKER_COOLDOWN` seconds, so games don't wait
  `AI_DECISION_TIMEOUT` for every move of a hanging AI. Load shedding (the
  in-flight cap, AI answers `overloaded` / `expired`) does not count.

## Limits and overload handling

The complete description — all layers, thresholds, what players see, tuning,
monitoring and measurements — is in
[chapter 9: Limits, load control and performance](09-limits-and-performance.md).
In short, for the server:

- **Rule**: running games always continue; new lobbies/games are refused when
  there is no capacity; abuse is rejected immediately.
- **Abuse limits** (`limits.py`, applied in `WizardService.handle_client`):
  connections in total and per IP (refused with `error` code `busy` + close
  1013), a token bucket per connection for messages (`rate_limited`, close
  1008 on floods), lobby creation per IP.
- **Load monitor** (`load.py`, `LoadMonitor.run()` in the app lifespan):
  samples event loop lag, CPU against the container quota, and cgroup
  throttling every second; levels `normal` / `busy` / `overloaded` with
  hysteresis.
- **Admission control** (`WizardService.accepting_games`): `create_lobby` and
  `start_game` are refused while overloaded or while `MAX_ACTIVE_GAMES` games
  run; everything else keeps working. Changes are broadcast to all clients as
  `server_status`.
- `/health` reports the load but its `status` stays "ok", so Docker never
  restarts an overloaded server (which would end all games).

## Configuration (`config.py`)

All values are read from environment variables when the module is imported.

| Variable | Default | Meaning |
|----------|---------|---------|
| `AI_SHARED_SECRET` | empty | secret the AI service must send; empty = no auth (dev only) |
| `AI_DECISION_TIMEOUT` | 5.0 s | wait for the AI before playing a random move |
| `AI_MOVE_DELAY` | 0.8 s | minimum time an AI move takes, so humans can follow |
| `TRICK_PAUSE` | 1.8 s | pause after a completed trick before AIs continue |
| `SESSION_TTL` | 1800 s | disconnected sessions are dropped after this |
| `MAX_LOBBIES` | 1000 | limit of simultaneously open lobbies |
| `ALLOWED_ORIGINS` | empty (= all) | comma separated allowed `Origin` headers for `/ws` |
| `LOG_LEVEL` | INFO | Python logging level |
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | 500 / 10 | open websocket connections |
| `MESSAGE_RATE` / `MESSAGE_BURST` | 10 / 30 | messages per second per connection |
| `MAX_RATE_VIOLATIONS` | 50 | dropped messages before a flooding connection is closed |
| `LOBBY_CREATE_RATE` | 10 | lobbies per IP per minute |
| `MAX_ACTIVE_GAMES` | 50 | games running at the same time |
| `LOAD_SAMPLE_INTERVAL` | 1.0 s | load sampling period |
| `LOAD_LAG_BUSY` / `LOAD_LAG_OVERLOAD` / `LOAD_LAG_SEVERE` | 0.1 / 0.25 / 1.0 s | event loop lag thresholds |
| `LOAD_CPU_BUSY` / `LOAD_CPU_OVERLOAD` | 0.7 / 0.9 | CPU share of the quota |
| `LOAD_THROTTLE_OVERLOAD` | 0.5 | throttled share of CPU periods |
| `LOAD_OVERLOAD_SAMPLES` / `LOAD_RECOVER_SAMPLES` | 3 / 10 | hysteresis |
| `LOAD_FORCE_LEVEL` | empty | pin the level (testing) |
| `AI_MAX_INFLIGHT` | 64 | waiting AI requests per AI connection (keep ≥ `MAX_ACTIVE_GAMES`) |
| `AI_BREAKER_FAILURES` / `AI_BREAKER_COOLDOWN` | 5 / 30 s | AI circuit breaker |

## Housekeeping

`WizardService.cleanup_loop()` runs every 30 s (started in the app lifespan)
and removes expired sessions (leaving their lobby first) and forgets idle
per-IP lobby-creation buckets.

## Tests (`wizard_server/tests/`)

Uses FastAPI's `TestClient` websockets with zero delays and a raised message
rate limit (`helpers.TEST_SETTINGS`).

- `test_server.py` — health, hello requirement, validation errors, the full
  lobby flow including a complete 20-round game between two humans and an AI,
  reconnect by token, leaving mid-game (seat becomes AI, host moves), the AI
  endpoint's secret and proxy checks, and a fake AI service.
- `test_protection.py` — message flood, connections per IP, lobby creation
  rate, forced overload, `MAX_ACTIVE_GAMES`, a running game continuing while
  overloaded, `server_status` messages.
- `test_limits.py` — unit tests of the token bucket, limiters, load levels,
  cgroup parsing and the AI circuit breaker / in-flight cap.

Fixtures (`conftest.py`): `client` (default test settings) and
`make_client(**overrides)` for custom settings. Enter websocket sessions with
`client.sockets.enter_context(client.websocket_connect(...))` (as
`helpers.connect` does), otherwise the test client hangs on shutdown.

## Limitations to be aware of

- In-memory state: restarting the server ends all lobbies and games.
- Must run as a single process (no `--workers`).
- No spectators: only seated players receive game state.
- A running game whose humans all disconnected keeps its `MAX_ACTIVE_GAMES`
  slot until their sessions expire (`SESSION_TTL`), because they may still
  reconnect. It costs no CPU while waiting.