Files
Wizard/docs/08-operations.md
T
marvinandClaude Opus 5.5 87dfcd55b7 Add resource limits, abuse protection and overload handling
Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.

- Docker: CPU/memory/pids caps per container (overridable in .env);
  uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
  rate per connection (token bucket, close 1008 on floods), lobby
  creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
  throttling, with hysteresis (normal/busy/overloaded, severe lag
  overloads immediately)
- Admission control: create_lobby and start_game refused while
  overloaded or MAX_ACTIVE_GAMES are running; server_status message
  broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
  requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
  games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
  limits; docs updated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 12:51:46 +02:00

11 KiB
Raw Blame History

8. Deployment, development & operations

Docker

Images

Service Dockerfile Build context Listens on Notes
wizard_website wizard_website/Dockerfile wizard_website/ 8082 (nginx) multi-stage: node build → nginx; build arg VITE_WS_URL (optional)
wizard_server wizard_server/Dockerfile wizard_server/ 8083 (uvicorn) installs requirements.txt, then game_logic/; runs as non-root; health check on /health
wizard_ai wizard_ai/Dockerfile wizard_ai/ plus named context game_logic – installs the game engine from the extra context; runs python -m wizard_ai serve

The AI image needs the game engine, which lives in the server folder. Compose passes it as an additional build context (requires docker compose ≥ 2.17):

build:
  context: ./wizard_ai
  additional_contexts:
    game_logic: ./wizard_server/game_logic

Building it by hand: docker build --build-context game_logic=wizard_server/game_logic wizard_ai.

Networks

Network Type Members
wizard_website external (shared with Caddy) wizard_website
wizard_api external (shared with Caddy) wizard_server
wizard_internal created by compose, internal: true (no internet) wizard_server, wizard_ai

No ports are published to the host; only Caddy reaches the containers.

Environment (.env next to compose.yaml)

Compose reads .env automatically; .env.example is only a template and is not read by anything.

Variable Used by Required Meaning
AI_SHARED_SECRET server + AI yes (compose refuses to start without it) shared secret for /internal/ai
ALLOWED_ORIGINS server no e.g. https://game.marvinkrausser.com
AI_AGENT AI no (default model) model, heuristic or random
LOG_LEVEL server + AI no INFO, DEBUG, …
SERVER_CPUS / SERVER_MEMORY compose no (1.0 / 512M) hard caps of the server container
AI_CPUS / AI_MEMORY compose no (1.0 / 1G) hard caps of the AI container
WEBSITE_CPUS / WEBSITE_MEMORY compose no (0.25 / 64M) hard caps of the website container
MAX_ACTIVE_GAMES server no (50) games running at the same time; more starts are refused ("server busy")
MAX_CONNECTIONS / MAX_CONNECTIONS_PER_IP server no (500 / 10) open websocket connections
AI_WORKERS / AI_MAX_QUEUE / AI_DEGRADE_QUEUE AI no (1 / 64 / 8) parallel decisions, waiting requests before refusing, queue depth from which the heuristic is used

How the limits work together is described in "Limits and overload handling" in the server chapter. Further server settings (AI_MOVE_DELAY, TRICK_PAUSE, load thresholds, …, see "Configuration" in the server chapter) can be added to the environment: section of wizard_server in compose.yaml.

AI models

./wizard_ai/models is mounted read-only to /srv/models; the service loads /srv/models/latest/ (a policy checkpoint directory). Put a new checkpoint there and restart wizard_ai — no rebuild needed. Without a checkpoint the heuristic agent is used.

Deploying

cp .env.example .env            # once; set AI_SHARED_SECRET
docker compose up -d --build    # build and (re)start everything
docker compose up -d --build wizard_website   # only one service
docker compose logs -f wizard_server
docker compose ps               # server shows "healthy" when /health answers

Reverse proxy (Caddy)

The existing Caddy container serves the game under game.marvinkrausser.com:

game.marvinkrausser.com {
    import common

    redir /wizard /wizard/website/ permanent
    redir /wizard/ /wizard/website/ permanent
    redir /wizard/website /wizard/website/ permanent

    @wizard_api path /wizard/api /wizard/api/*
    handle @wizard_api {
        uri strip_prefix /wizard/api
        reverse_proxy wizard_server:8083
    }

    handle_path /wizard/website/* {
        reverse_proxy wizard_website:8082
    }

    handle {
        respond 404
    }
}

What each part means for the app:

  • The website must be opened with a trailing slash (/wizard/website/) so relative asset paths resolve — the redirects take care of that.
  • Both prefixes are stripped, so the containers do not know about /wizard/…. The frontend finds the API through the relative URL ../api/ws.
  • Websockets are proxied automatically by Caddy.
  • /wizard/api/internal/ai is reachable through this block, but the server refuses it because the request carries Caddy's forwarding headers.

URLs: game https://game.marvinkrausser.com/wizard/website/, health check https://game.marvinkrausser.com/wizard/api/health.

Local development

One Python virtual environment for server and AI:

python -m venv .venv && . .venv/bin/activate
pip install -r wizard_server/requirements-dev.txt -r wizard_ai/requirements-dev.txt

requirements-dev.txt installs the game engine editable (-e), so changes to wizard_game are picked up immediately.

Run the three parts in three terminals:

# 1. server on :8000 (add AI_MOVE_DELAY=0 TRICK_PAUSE=0 to make AI players instant)
cd wizard_server && AI_SHARED_SECRET=dev uvicorn app.main:app --reload

# 2. AI service
cd wizard_ai && AI_SHARED_SECRET=dev SERVER_URL=ws://localhost:8000/internal/ai python -m wizard_ai serve

# 3. website on :5173 (proxies /api → localhost:8000 without the /api prefix)
cd wizard_website && npm install && npm run dev

Open http://localhost:5173. To play as several humans, use several browser tabs (each tab is its own session). Without step 2 the game still works; AI seats then play random moves.

Tests and checks

cd wizard_server && pytest      # game engine (game_logic/tests) + server (tests/)
cd wizard_ai && pytest
cd wizard_website && npm run lint && npm run build
Suite Count Covers
game engine 52 rules, full random games for 3–6 players, invariants, JSON
server 32 websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels
AI 17 encoding, agents, env, trainer, checkpoints, service limits

Troubleshooting

Symptom Cause / fix
Browser console: Refused to apply style … MIME type (''), 404 for /assets/… The page loads assets from the wrong path. The build must use relative paths (base: './' in vite.config.ts) and the page must be opened with a trailing slash. Rebuild the website image after changing the Vite config.
Red banner "Connection lost. Reconnecting…" forever The websocket URL does not reach the server. Check https://…/wizard/api/health; check that the Caddy block strips /wizard/api; check ALLOWED_ORIGINS (must contain the exact page origin, e.g. https://game.marvinkrausser.com).
Banner "This game was opened in another window" The same session (token) was opened in a second tab; reload this tab to take it back.
Lobby shows "The AI service is offline" wizard_ai is not connected. docker compose logs wizard_ai: Server rejected the connection, check AI_SHARED_SECRET → secrets differ; connection refused → server not running or wrong SERVER_URL.
AI players only make weird/random moves Same as above (random fallback), or the AI answers too slowly (AI_DECISION_TIMEOUT), see server logs "Playing random move for player …".
Game waits and nothing happens It is a disconnected human's turn (seat shows "offline"). The game waits until they reconnect or their session expires (SESSION_TTL, then an AI takes over).
All lobbies gone The server restarted — state is in memory only.
Start page / lobby says "The server is busy", Create/Start disabled The server is overloaded or MAX_ACTIVE_GAMES games are running. Check https://…/wizard/api/health (load, cpu, throttled, active_games) and docker stats. Running games are not affected. If it happens often, raise SERVER_CPUS or MAX_ACTIVE_GAMES.
Banner "The server is full right now" Connection limit reached (MAX_CONNECTIONS, or MAX_CONNECTIONS_PER_IP — e.g. many tabs behind one NAT). The page retries on its own every 15–60 s.
Error toast "Too many messages, slow down." A client exceeded the message rate limit (10/s). Normal play never gets near it; a script or broken client might.
AI plays noticeably worse under load The AI service switched to the heuristic because it was busy (AI_DEGRADE_QUEUE) or the model was too slow; see docker compose logs wizard_ai ("Busy … using heuristic agent"). Raise AI_CPUS/AI_WORKERS or use a smaller model.
Tests pass but behaviour does not change after editing wizard_game A non-editable copy of wizard_game is installed in your virtualenv and shadows the source. pip uninstall wizard-game && pip install -e wizard_server/game_logic. Check with python -c "import wizard_game; print(wizard_game.__file__)".
Server tests hang A test opened a websocket without a context manager; use client.sockets.enter_context(client.websocket_connect(...)).

Recipes

Add a new game option (e.g. a rule variant)

  1. wizard_game/state.py: add a field to GameOptions and to to_dict / from_dict.
  2. wizard_game/game.py: implement the behaviour using self.options.<field>; add tests in game_logic/tests/test_game.py.
  3. wizard_server/app/protocol.py: add the field to SetOptions; service.py: pass it into GameOptions(...).
  4. wizard_website/src/api/types.ts: add it to GameOptions and the set_options message; LobbyScreen.tsx: add a checkbox (copy the "Hidden bids" one).
  5. wizard_ai/wizard_ai/model/encoding.py: add a feature (and update FEATURE_SIZE) if the AI should know about it; __main__.py: add a --flag to train.

Add a new client message

  1. protocol.py: new pydantic model, add it to the ClientMessage union.
  2. service.py: handle it in _dispatch (usually by calling a new Lobby method and then lobby.broadcast()).
  3. types.ts: add it to ClientMessage; useWizard.ts: add a function to actions.
  4. Add a test in wizard_server/tests/test_server.py.

Check resource usage

docker stats wizard-wizard_server-1 wizard-wizard_ai-1 wizard-wizard_website-1
curl -s https://game.marvinkrausser.com/wizard/api/health

To test the overload behaviour without real load, start the server with LOAD_FORCE_LEVEL=overloaded (new games are refused, the UI shows the busy notice).

Change how fast AI players move

Set AI_MOVE_DELAY (per move) and TRICK_PAUSE (after a trick) for wizard_server in compose.yaml.

Add another AI agent

Subclass Agent in wizard_ai/wizard_ai/agents/, add it to AGENT_NAMES and make_agent in agents/__init__.py, then use it via AI_AGENT=<name> or in evaluate --agents.

Change container ports

Website: listen in wizard_website/nginx.conf and EXPOSE in its Dockerfile. Server: --port, EXPOSE and the health check URL in wizard_server/Dockerfile, and SERVER_URL of wizard_ai in compose.yaml. Then update the Caddy block.