Add resource limits, abuse protection and overload handling

Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.

- Docker: CPU/memory/pids caps per container (overridable in .env);
  uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
  rate per connection (token bucket, close 1008 on floods), lobby
  creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
  throttling, with hysteresis (normal/busy/overloaded, severe lag
  overloads immediately)
- Admission control: create_lobby and start_game refused while
  overloaded or MAX_ACTIVE_GAMES are running; server_status message
  broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
  requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
  games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
  limits; docs updated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-27 12:51:46 +02:00
co-authored by Claude Opus 5.5
parent 905574e8f1
commit 87dfcd55b7
33 changed files with 1591 additions and 128 deletions
+26 -3
View File
@@ -44,8 +44,16 @@ not read by anything.
| `ALLOWED_ORIGINS` | server | no | e.g. `https://game.marvinkrausser.com` |
| `AI_AGENT` | AI | no (default `model`) | `model`, `heuristic` or `random` |
| `LOG_LEVEL` | server + AI | no | `INFO`, `DEBUG`, … |
| `SERVER_CPUS` / `SERVER_MEMORY` | compose | no (1.0 / 512M) | hard caps of the server container |
| `AI_CPUS` / `AI_MEMORY` | compose | no (1.0 / 1G) | hard caps of the AI container |
| `WEBSITE_CPUS` / `WEBSITE_MEMORY` | compose | no (0.25 / 64M) | hard caps of the website container |
| `MAX_ACTIVE_GAMES` | server | no (50) | games running at the same time; more starts are refused ("server busy") |
| `MAX_CONNECTIONS` / `MAX_CONNECTIONS_PER_IP` | server | no (500 / 10) | open websocket connections |
| `AI_WORKERS` / `AI_MAX_QUEUE` / `AI_DEGRADE_QUEUE` | AI | no (1 / 64 / 8) | parallel decisions, waiting requests before refusing, queue depth from which the heuristic is used |
Further server settings (`AI_MOVE_DELAY`, `TRICK_PAUSE`, …, see
How the limits work together is described in "Limits and overload handling"
in [the server chapter](04-server.md). Further server settings
(`AI_MOVE_DELAY`, `TRICK_PAUSE`, load thresholds, …, see
"Configuration" in [the server chapter](04-server.md)) can be added to
the `environment:` section of `wizard_server` in `compose.yaml`.
@@ -147,8 +155,8 @@ cd wizard_website && npm run lint && npm run build
| Suite | Count | Covers |
|-------|-------|--------|
| game engine | 52 | rules, full random games for 3–6 players, invariants, JSON |
| server | 9 | websocket flows incl. a complete game, reconnect, AI endpoint security |
| AI | 13 | encoding, agents, env, trainer, checkpoints |
| server | 32 | websocket flows incl. a complete game, reconnect, AI endpoint security, abuse limits, overload handling, load levels |
| AI | 17 | encoding, agents, env, trainer, checkpoints, service limits |
## Troubleshooting
@@ -161,6 +169,10 @@ cd wizard_website && npm run lint && npm run build
| AI players only make weird/random moves | Same as above (random fallback), or the AI answers too slowly (`AI_DECISION_TIMEOUT`), see server logs "Playing random move for player …". |
| Game waits and nothing happens | It is a disconnected human's turn (seat shows "offline"). The game waits until they reconnect or their session expires (`SESSION_TTL`, then an AI takes over). |
| All lobbies gone | The server restarted — state is in memory only. |
| Start page / lobby says "The server is busy", Create/Start disabled | The server is overloaded or `MAX_ACTIVE_GAMES` games are running. Check `https://…/wizard/api/health` (`load`, `cpu`, `throttled`, `active_games`) and `docker stats`. Running games are not affected. If it happens often, raise `SERVER_CPUS` or `MAX_ACTIVE_GAMES`. |
| Banner "The server is full right now" | Connection limit reached (`MAX_CONNECTIONS`, or `MAX_CONNECTIONS_PER_IP` — e.g. many tabs behind one NAT). The page retries on its own every 15–60 s. |
| Error toast "Too many messages, slow down." | A client exceeded the message rate limit (10/s). Normal play never gets near it; a script or broken client might. |
| AI plays noticeably worse under load | The AI service switched to the heuristic because it was busy (`AI_DEGRADE_QUEUE`) or the model was too slow; see `docker compose logs wizard_ai` ("Busy … using heuristic agent"). Raise `AI_CPUS`/`AI_WORKERS` or use a smaller model. |
| Tests pass but behaviour does not change after editing `wizard_game` | A non-editable copy of `wizard_game` is installed in your virtualenv and shadows the source. `pip uninstall wizard-game && pip install -e wizard_server/game_logic`. Check with `python -c "import wizard_game; print(wizard_game.__file__)"`. |
| Server tests hang | A test opened a websocket without a context manager; use `client.sockets.enter_context(client.websocket_connect(...))`. |
@@ -190,6 +202,17 @@ cd wizard_website && npm run lint && npm run build
`actions`.
4. Add a test in `wizard_server/tests/test_server.py`.
### Check resource usage
```bash
docker stats wizard-wizard_server-1 wizard-wizard_ai-1 wizard-wizard_website-1
curl -s https://game.marvinkrausser.com/wizard/api/health
```
To test the overload behaviour without real load, start the server with
`LOAD_FORCE_LEVEL=overloaded` (new games are refused, the UI shows the busy
notice).
### Change how fast AI players move
Set `AI_MOVE_DELAY` (per move) and `TRICK_PAUSE` (after a trick) for