Add resource limits, abuse protection and overload handling

Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.

- Docker: CPU/memory/pids caps per container (overridable in .env);
  uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
  rate per connection (token bucket, close 1008 on floods), lobby
  creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
  throttling, with hysteresis (normal/busy/overloaded, severe lag
  overloads immediately)
- Admission control: create_lobby and start_game refused while
  overloaded or MAX_ACTIVE_GAMES are running; server_status message
  broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
  requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
  games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
  limits; docs updated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-27 12:51:46 +02:00
co-authored by Claude Opus 5.5
parent 905574e8f1
commit 87dfcd55b7
33 changed files with 1591 additions and 128 deletions
+36 -5
View File
@@ -93,25 +93,51 @@ contains the **complete** current state.
- `ai_available` is `false` when no AI service is connected; AI seats then play
random moves.
#### `server_status`
Sent right after `welcome` and to every connected client whenever the server's
load level or capacity changes.
```json
{"type": "server_status", "load": "overloaded", "accepting_games": false}
```
- `load`: `normal`, `busy` (informational) or `overloaded`.
- `accepting_games`: `false` while the server refuses `create_lobby` and
`start_game` (overloaded, or `MAX_ACTIVE_GAMES` games running). Everything
else, including running games, keeps working.
#### `error`
```json
{"type": "error", "message": "Only the host can do that"}
{"type": "error", "message": "The server is busy right now, …", "code": "busy"}
```
`code` is present for errors the client may react to:
| `code` | Meaning |
|--------|---------|
| `busy` | no capacity: connection limit reached (followed by close 1013), or `create_lobby` / `start_game` refused because the server does not accept new games |
| `rate_limited` | too many messages on this connection, or too many lobbies created from this IP |
Sent only to the client that caused it. Typical messages: `Send 'hello' first`,
`Invalid message: …` (followed by the pydantic reason),
`Lobby ABC123 does not exist`, `The game in this lobby has already started`,
`This lobby is full`, `Only the host can do that`,
`Wizard needs 3 to 6 players`, `Illegal action Bid(value=2) for player 0`,
`It is not player 1's turn`, `You were removed from the lobby`,
`This session was opened in another window`.
`This session was opened in another window`, `Too many messages, slow down.`,
`You are creating lobbies too quickly, please wait a moment.`
### Close codes
| Code | Meaning | Frontend reaction |
|------|---------|-------------------|
| (handshake rejected) | origin not in `ALLOWED_ORIGINS`; the server closes with 1008 before accepting, so the browser sees a failed connection | reconnects with backoff |
| 1008 | message flood (too many dropped messages) | reconnects with backoff |
| 1009 | message larger than 64 KB (uvicorn `--ws-max-size`) | reconnects with backoff |
| 1013 | "Try Again Later": connection limit reached (preceded by an `error` with code `busy`) | banner "The server is full", retries slowly (15 s, 30 s, 60 s …) |
| 4000 | the session was opened in another tab | stops reconnecting, shows a banner |
| other | network problem, server restart | reconnects (1 s, 2 s, 4 s … max 10 s) |
@@ -153,9 +179,9 @@ headers (see "Routes" in [the server chapter](04-server.md)).
| Direction | Message | Meaning |
|-----------|---------|---------|
| AI → server | `{"type":"hello","agent":"heuristic"}` | informational, logged |
| server → AI | `{"type":"decide","request_id":"<hex>","observation":{…}}` | choose an action for `observation.player` |
| AI → server | `{"type":"action","request_id":"<same>","action":{"type":"play","card":17}}` | the chosen action |
| AI → server | `{"type":"error","request_id":"<same>","message":"…"}` | the AI failed; the server plays a random move |
| server → AI | `{"type":"decide","request_id":"<hex>","timeout":5.0,"observation":{…}}` | choose an action for `observation.player`; the server waits at most `timeout` seconds |
| AI → server | `{"type":"action","request_id":"<same>","action":{"type":"play","card":17},"agent":"heuristic"}` | the chosen action; `agent` names the agent that decided (logged) |
| AI → server | `{"type":"error","request_id":"<same>","message":"…"}` | the AI failed or refused (`overloaded`, `expired`); the server plays a random move |
Rules:
@@ -163,6 +189,11 @@ Rules:
(matched by `request_id`) and may handle several concurrently.
- The answer must be one of `observation.legal_actions`; otherwise the server
plays a random legal move.
- No answer within `AI_DECISION_TIMEOUT` (5 s) → random move.
- No answer within `AI_DECISION_TIMEOUT` (5 s) → random move. The AI service
drops requests whose `timeout` passed before it could start them (`expired`).
- The server stops asking an AI connection that already has `AI_MAX_INFLIGHT`
unanswered requests, and pauses the AI after repeated failures (circuit
breaker); see "Limits and overload handling" in
[the server chapter](04-server.md).
- Several AI service instances may connect; the server distributes requests
round robin.