Wizard/docs/08-operations.md

# 8. Deployment, development & operations

## Docker

### Images

| Service | Dockerfile | Build context | Listens on | Notes |
|---------|-----------|---------------|------------|-------|
| `wizard_website` | `wizard_website/Dockerfile` | `wizard_website/` | 8082 (nginx) | multi-stage: node build → nginx; build arg `VITE_WS_URL` (optional) |
| `wizard_server` | `wizard_server/Dockerfile` | `wizard_server/` | 8083 (uvicorn) | installs `requirements.txt`, then `game_logic/`; runs as non-root; health check on `/health` |
| `wizard_ai` | `wizard_ai/Dockerfile` | `wizard_ai/` **plus** named context `game_logic` | – | installs the game engine from the extra context; runs `python -m wizard_ai serve` |

The AI image needs the game engine, which lives in the server folder. Compose
passes it as an additional build context (requires docker compose ≥ 2.17):

```yaml
build:
  context: ./wizard_ai
  additional_contexts:
    game_logic: ./wizard_server/game_logic
```

Building it by hand: `docker build --build-context
game_logic=wizard_server/game_logic wizard_ai`.

### Networks

| Network | Type | Members |
|---------|------|---------|
| `wizard_website` | external (shared with Caddy) | `wizard_website` |
| `wizard_api` | external (shared with Caddy) | `wizard_server` |
| `wizard_internal` | created by compose, `internal: true` (no internet) | `wizard_server`, `wizard_ai` |

No ports are published to the host; only Caddy reaches the containers.

### Environment (`.env` next to `compose.yaml`)

Compose reads `.env` automatically; `.env.example` is only a template and is
not read by anything.

| Variable | Used by | Required | Meaning |
|----------|---------|----------|---------|
| `AI_SHARED_SECRET` | server + AI | **yes** (compose refuses to start without it) | shared secret for `/internal/ai` |
| `ALLOWED_ORIGINS` | server | no | e.g. `https://game.marvinkrausser.com` |
| `AI_AGENT` | AI | no (default `model`) | `model`, `heuristic` or `random` |
| `LOG_LEVEL` | server + AI | no | `INFO`, `DEBUG`, … |
| `SERVER_CPUS` / `SERVER_MEMORY` | compose | no (1.0 / 512M) | hard caps of the server container |
| `AI_CPUS` / `AI_MEMORY` | compose | no (1.0 / 1G) | hard caps of the AI container |
| `WEBSITE_CPUS` / `WEBSITE_MEMORY` | compose | no (0.25 / 64M) | hard caps of the website container |
| `MAX_ACTIVE_GAMES` | server | no (50) | games running at the same time; more starts are refused ("server busy") |
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | no (500 / 10) | open websocket connections |
| `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | no (1 / 64 / 8) | parallel decisions, waiting requests before refusing, queue depth from which the heuristic is used |

How the limits work together, the full list of settings, measurements and a
tuning guide are in [chapter 9](09-limits-and-performance.md). Further server settings
(`AI_MOVE_DELAY`, `TRICK_PAUSE`, load thresholds, …, see
"Configuration" in [the server chapter](04-server.md)) can be added to
the `environment:` section of `wizard_server` in `compose.yaml`.

### AI models

`./wizard_ai/models` is mounted read-only to `/srv/models`; the service loads
`/srv/models/latest/` (a policy checkpoint directory). Put a new checkpoint
there and restart `wizard_ai` — no rebuild needed. Without a checkpoint the
heuristic agent is used.

### Deploying

```bash
cp .env.example .env            # once; set AI_SHARED_SECRET
docker compose up -d --build    # build and (re)start everything
docker compose up -d --build wizard_website   # only one service
docker compose logs -f wizard_server
docker compose ps               # server shows "healthy" when /health answers
```

## Reverse proxy (Caddy)

The existing Caddy container serves the game under `game.marvinkrausser.com`:

```caddy
game.marvinkrausser.com {
    import common

    redir /wizard /wizard/website/ permanent
    redir /wizard/ /wizard/website/ permanent
    redir /wizard/website /wizard/website/ permanent

    @wizard_api path /wizard/api /wizard/api/*
    handle @wizard_api {
        uri strip_prefix /wizard/api
        reverse_proxy wizard_server:8083
    }

    handle_path /wizard/website/* {
        reverse_proxy wizard_website:8082
    }

    handle {
        respond 404
    }
}
```

What each part means for the app:

- The website must be opened with a trailing slash (`/wizard/website/`) so
  relative asset paths resolve — the redirects take care of that.
- Both prefixes are stripped, so the containers do not know about `/wizard/…`.
  The frontend finds the API through the relative URL `../api/ws`.
- Websockets are proxied automatically by Caddy.
- `/wizard/api/internal/ai` is reachable through this block, but the server
  refuses it because the request carries Caddy's forwarding headers.

URLs: game `https://game.marvinkrausser.com/wizard/website/`, health check
`https://game.marvinkrausser.com/wizard/api/health`.

## Local development

One Python virtual environment for server and AI:

```bash
python -m venv .venv && . .venv/bin/activate
pip install -r wizard_server/requirements-dev.txt -r wizard_ai/requirements-dev.txt
```

`requirements-dev.txt` installs the game engine **editable** (`-e`), so changes
to `wizard_game` are picked up immediately.

Run the three parts in three terminals:

```bash
# 1. server on :8000 (add AI_MOVE_DELAY=0 TRICK_PAUSE=0 to make AI players instant)
cd wizard_server && AI_SHARED_SECRET=dev uvicorn app.main:app --reload

# 2. AI service
cd wizard_ai && AI_SHARED_SECRET=dev SERVER_URL=ws://localhost:8000/internal/ai python -m wizard_ai serve

# 3. website on :5173 (proxies /api → localhost:8000 without the /api prefix)
cd wizard_website && npm install && npm run dev
```

Open <http://localhost:5173>. To play as several humans, use several browser
tabs (each tab is its own session). Without step 2 the game still works; AI
seats then play random moves.

## Tests and checks

```bash
cd wizard_server && pytest      # game engine (game_logic/tests) + server (tests/)
cd wizard_ai && pytest
cd wizard_website && npm run lint && npm run build
```

| Suite | Count | Covers |
|-------|-------|--------|
| game engine | 52 | rules, full random games for 3–6 players, invariants, JSON |
| server | 33 | websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels |
| AI | 17 | encoding, agents, env, trainer, checkpoints, service limits |

## Troubleshooting

| Symptom | Cause / fix |
|---------|-------------|
| Browser console: *Refused to apply style … MIME type ('')*, 404 for `/assets/…` | The page loads assets from the wrong path. The build must use relative paths (`base: './'` in `vite.config.ts`) and the page must be opened with a trailing slash. Rebuild the website image after changing the Vite config. |
| Red banner "Connection lost. Reconnecting…" forever | The websocket URL does not reach the server. Check `https://…/wizard/api/health`; check that the Caddy block strips `/wizard/api`; check `ALLOWED_ORIGINS` (must contain the exact page origin, e.g. `https://game.marvinkrausser.com`). |
| Banner "This game was opened in another window" | The same session (token) was opened in a second tab; reload this tab to take it back. |
| Lobby shows "The AI service is offline" | `wizard_ai` is not connected. `docker compose logs wizard_ai`: *Server rejected the connection, check AI_SHARED_SECRET* → secrets differ; connection refused → server not running or wrong `SERVER_URL`. |
| AI players only make weird/random moves | Same as above (random fallback), or the AI answers too slowly (`AI_DECISION_TIMEOUT`), see server logs "Playing random move for player …". |
| Game waits and nothing happens | It is a disconnected human's turn (seat shows "offline"). The game waits until they reconnect or their session expires (`SESSION_TTL`, then an AI takes over). |
| All lobbies gone | The server restarted — state is in memory only. |
| Start page / lobby says "The server is busy", Create/Start disabled | The server is overloaded or `MAX_ACTIVE_GAMES` games are running. Check `https://…/wizard/api/health` (`load`, `cpu`, `throttled`, `active_games`) and `docker stats`. Running games are not affected. If it happens often, raise `SERVER_CPUS` or `MAX_ACTIVE_GAMES`. |
| Banner "The server is full right now" | Connection limit reached (`MAX_CONNECTIONS`, or `MAX_CONNECTIONS_PER_IP` — e.g. many tabs behind one NAT). The page retries on its own every 15–60 s. |
| Error toast "Too many messages, slow down." | A client exceeded the message rate limit (10/s). Normal play never gets near it; a script or broken client might. |
| AI plays noticeably worse under load | The AI service switched to the heuristic because it was busy (`AI_DEGRADE_QUEUE`) or the model was too slow; see `docker compose logs wizard_ai` ("Busy … using heuristic agent"). Raise `AI_CPUS`/`AI_WORKERS` or use a smaller model. |
| Tests pass but behaviour does not change after editing `wizard_game` | A non-editable copy of `wizard_game` is installed in your virtualenv and shadows the source. `pip uninstall wizard-game && pip install -e wizard_server/game_logic`. Check with `python -c "import wizard_game; print(wizard_game.__file__)"`. |
| Server tests hang | A test opened a websocket without a context manager; use `client.sockets.enter_context(client.websocket_connect(...))`. |

## Recipes

### Add a new game option (e.g. a rule variant)

1. `wizard_game/state.py`: add a field to `GameOptions` and to `to_dict` /
   `from_dict`.
2. `wizard_game/game.py`: implement the behaviour using
   `self.options.<field>`; add tests in `game_logic/tests/test_game.py`.
3. `wizard_server/app/protocol.py`: add the field to `SetOptions`;
   `service.py`: pass it into `GameOptions(...)`.
4. `wizard_website/src/api/types.ts`: add it to `GameOptions` and the
   `set_options` message; `LobbyScreen.tsx`: add a checkbox (copy the
   "Hidden bids" one).
5. `wizard_ai/wizard_ai/model/encoding.py`: add a feature (and update
   `FEATURE_SIZE`) if the AI should know about it; `__main__.py`: add a
   `--flag` to `train`.

### Add a new client message

1. `protocol.py`: new pydantic model, add it to the `ClientMessage` union.
2. `service.py`: handle it in `_dispatch` (usually by calling a new `Lobby`
   method and then `lobby.broadcast()`).
3. `types.ts`: add it to `ClientMessage`; `useWizard.ts`: add a function to
   `actions`.
4. Add a test in `wizard_server/tests/test_server.py`.

### Check resource usage

```bash
docker stats wizard-wizard_server-1 wizard-wizard_ai-1 wizard-wizard_website-1
curl -s https://game.marvinkrausser.com/wizard/api/health
```

To test the overload behaviour without real load, start the server with
`LOAD_FORCE_LEVEL=overloaded` (new games are refused, the UI shows the busy
notice).

### Change how fast AI players move

Set `AI_MOVE_DELAY` (per move) and `TRICK_PAUSE` (after a trick) for
`wizard_server` in `compose.yaml`.

### Add another AI agent

Subclass `Agent` in `wizard_ai/wizard_ai/agents/`, add it to `AGENT_NAMES`
and `make_agent` in `agents/__init__.py`, then use it via `AI_AGENT=<name>` or
in `evaluate --agents`.

### Change container ports

Website: `listen` in `wizard_website/nginx.conf` and `EXPOSE` in its
Dockerfile. Server: `--port`, `EXPOSE` and the health check URL in
`wizard_server/Dockerfile`, and `SERVER_URL` of `wizard_ai` in `compose.yaml`.
Then update the Caddy block.