Add resource limits, abuse protection and overload handling
Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.
- Docker: CPU/memory/pids caps per container (overridable in .env);
uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
rate per connection (token bucket, close 1008 on floods), lobby
creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
throttling, with hysteresis (normal/busy/overloaded, severe lag
overloads immediately)
- Admission control: create_lobby and start_game refused while
overloaded or MAX_ACTIVE_GAMES are running; server_status message
broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
limits; docs updated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+36
-5
@@ -93,25 +93,51 @@ contains the **complete** current state.
|
||||
- `ai_available` is `false` when no AI service is connected; AI seats then play
|
||||
random moves.
|
||||
|
||||
#### `server_status`
|
||||
|
||||
Sent right after `welcome` and to every connected client whenever the server's
|
||||
load level or capacity changes.
|
||||
|
||||
```json
|
||||
{"type": "server_status", "load": "overloaded", "accepting_games": false}
|
||||
```
|
||||
|
||||
- `load`: `normal`, `busy` (informational) or `overloaded`.
|
||||
- `accepting_games`: `false` while the server refuses `create_lobby` and
|
||||
`start_game` (overloaded, or `MAX_ACTIVE_GAMES` games running). Everything
|
||||
else, including running games, keeps working.
|
||||
|
||||
#### `error`
|
||||
|
||||
```json
|
||||
{"type": "error", "message": "Only the host can do that"}
|
||||
{"type": "error", "message": "The server is busy right now, …", "code": "busy"}
|
||||
```
|
||||
|
||||
`code` is present for errors the client may react to:
|
||||
|
||||
| `code` | Meaning |
|
||||
|--------|---------|
|
||||
| `busy` | no capacity: connection limit reached (followed by close 1013), or `create_lobby` / `start_game` refused because the server does not accept new games |
|
||||
| `rate_limited` | too many messages on this connection, or too many lobbies created from this IP |
|
||||
|
||||
Sent only to the client that caused it. Typical messages: `Send 'hello' first`,
|
||||
`Invalid message: …` (followed by the pydantic reason),
|
||||
`Lobby ABC123 does not exist`, `The game in this lobby has already started`,
|
||||
`This lobby is full`, `Only the host can do that`,
|
||||
`Wizard needs 3 to 6 players`, `Illegal action Bid(value=2) for player 0`,
|
||||
`It is not player 1's turn`, `You were removed from the lobby`,
|
||||
`This session was opened in another window`.
|
||||
`This session was opened in another window`, `Too many messages, slow down.`,
|
||||
`You are creating lobbies too quickly, please wait a moment.`
|
||||
|
||||
### Close codes
|
||||
|
||||
| Code | Meaning | Frontend reaction |
|
||||
|------|---------|-------------------|
|
||||
| (handshake rejected) | origin not in `ALLOWED_ORIGINS`; the server closes with 1008 before accepting, so the browser sees a failed connection | reconnects with backoff |
|
||||
| 1008 | message flood (too many dropped messages) | reconnects with backoff |
|
||||
| 1009 | message larger than 64 KB (uvicorn `--ws-max-size`) | reconnects with backoff |
|
||||
| 1013 | "Try Again Later": connection limit reached (preceded by an `error` with code `busy`) | banner "The server is full", retries slowly (15 s, 30 s, 60 s …) |
|
||||
| 4000 | the session was opened in another tab | stops reconnecting, shows a banner |
|
||||
| other | network problem, server restart | reconnects (1 s, 2 s, 4 s … max 10 s) |
|
||||
|
||||
@@ -153,9 +179,9 @@ headers (see "Routes" in [the server chapter](04-server.md)).
|
||||
| Direction | Message | Meaning |
|
||||
|-----------|---------|---------|
|
||||
| AI → server | `{"type":"hello","agent":"heuristic"}` | informational, logged |
|
||||
| server → AI | `{"type":"decide","request_id":"<hex>","observation":{…}}` | choose an action for `observation.player` |
|
||||
| AI → server | `{"type":"action","request_id":"<same>","action":{"type":"play","card":17}}` | the chosen action |
|
||||
| AI → server | `{"type":"error","request_id":"<same>","message":"…"}` | the AI failed; the server plays a random move |
|
||||
| server → AI | `{"type":"decide","request_id":"<hex>","timeout":5.0,"observation":{…}}` | choose an action for `observation.player`; the server waits at most `timeout` seconds |
|
||||
| AI → server | `{"type":"action","request_id":"<same>","action":{"type":"play","card":17},"agent":"heuristic"}` | the chosen action; `agent` names the agent that decided (logged) |
|
||||
| AI → server | `{"type":"error","request_id":"<same>","message":"…"}` | the AI failed or refused (`overloaded`, `expired`); the server plays a random move |
|
||||
|
||||
Rules:
|
||||
|
||||
@@ -163,6 +189,11 @@ Rules:
|
||||
(matched by `request_id`) and may handle several concurrently.
|
||||
- The answer must be one of `observation.legal_actions`; otherwise the server
|
||||
plays a random legal move.
|
||||
- No answer within `AI_DECISION_TIMEOUT` (5 s) → random move.
|
||||
- No answer within `AI_DECISION_TIMEOUT` (5 s) → random move. The AI service
|
||||
drops requests whose `timeout` passed before it could start them (`expired`).
|
||||
- The server stops asking an AI connection that already has `AI_MAX_INFLIGHT`
|
||||
unanswered requests, and pauses the AI after repeated failures (circuit
|
||||
breaker); see "Limits and overload handling" in
|
||||
[the server chapter](04-server.md).
|
||||
- Several AI service instances may connect; the server distributes requests
|
||||
round robin.
|
||||
|
||||
Reference in New Issue
Block a user