Add resource limits, abuse protection and overload handling

Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.

- Docker: CPU/memory/pids caps per container (overridable in .env);
  uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
  rate per connection (token bucket, close 1008 on floods), lobby
  creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
  throttling, with hysteresis (normal/busy/overloaded, severe lag
  overloads immediately)
- Admission control: create_lobby and start_game refused while
  overloaded or MAX_ACTIVE_GAMES are running; server_status message
  broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
  requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
  games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
  limits; docs updated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-27 12:51:46 +02:00
co-authored by Claude Opus 5.5
parent 905574e8f1
commit 87dfcd55b7
33 changed files with 1591 additions and 128 deletions
+28 -8
View File
@@ -194,8 +194,11 @@ registered with `@register_policy`, and calls its `load`.
the container (`compose.yaml`); restart `wizard_ai`. With `AGENT=model`
(default) it is loaded automatically.
Model size matters: the service answers every AI move in a worker thread and
the server waits at most `AI_DECISION_TIMEOUT` (5 s).
Model size matters: the service answers every AI move in a worker thread, the
container is capped at `AI_CPUS` (1 CPU by default) and the server waits at
most `AI_DECISION_TIMEOUT` (5 s). If the model is too slow the service
automatically answers with the heuristic instead (see below), so measure the
decision time before deploying.
## Training (`training/`)
@@ -244,14 +247,28 @@ count as a win for everyone tied). Seeded for fair comparisons.
## Production service (`service/client.py`)
`AIService(server_url, agent, secret).run_forever()`:
`AIService(server_url, agent, secret, fallback=…, workers=…, max_queue=…, degrade_queue=…).run_forever()`:
- connects to `SERVER_URL` with `Authorization: Bearer <AI_SHARED_SECRET>`,
sends `hello`, then answers `decide` requests (protocol in
[chapter 5](05-protocol.md#ai-protocol));
- each request runs `agent.act` in a worker thread (`asyncio.to_thread`) behind
a lock, so a slow model does not block the websocket and non-thread-safe
agents stay consistent;
- each request runs `agent.act` in a worker thread (`asyncio.to_thread`), so a
slow model does not block the websocket;
- **bounded work** (`handle_request`):
- at most `AI_WORKERS` (default 1) decisions are computed at the same time —
agents must be thread safe if you raise it;
- at most `AI_MAX_QUEUE` (64) requests may be in progress; more are answered
with `error` "overloaded" immediately;
- a request whose server `timeout` passed while it waited for a worker is
answered with `error` "expired" without computing it;
- **degradation**: when `AI_DEGRADE_QUEUE` (8) or more other requests are
waiting, or the primary agent's average decision time is more than half of
the remaining time, the cheap fallback agent (`HeuristicAgent`) decides
instead. `serve` sets the heuristic as fallback automatically unless the
primary agent already is the heuristic. The reply's `agent` field says which
one decided;
- `OMP_NUM_THREADS` / `MKL_NUM_THREADS` are set to `AI_WORKERS` in
`compose.yaml`, so numeric libraries don't start a thread per host core;
- errors while deciding are reported back as `error` (the server then plays a
random move) instead of crashing;
- reconnects forever with exponential backoff (1 s … 30 s); logs a hint if the
@@ -271,10 +288,13 @@ Environment variables (`config.py`): `SERVER_URL`
(default `ws://localhost:8000/internal/ai`; in Docker
`ws://wizard_server:8083/internal/ai`), `AI_SHARED_SECRET`, `AGENT`
(default `model`), `MODEL_PATH` (default `models/latest`, in Docker
`/srv/models/latest`), `LOG_LEVEL`.
`/srv/models/latest`), `LOG_LEVEL`, `AI_WORKERS` (1), `AI_MAX_QUEUE` (64),
`AI_DEGRADE_QUEUE` (8).
## Tests (`wizard_ai/tests/test_ai.py`)
Feature length and masks for 3–6 players, action-space round trip, all agents
play legal full games, env rewards add up to scores, trainer runs and writes a
loadable checkpoint, `make_agent` fallback, heuristic beats random.
loadable checkpoint, `make_agent` fallback, heuristic beats random, and the
service limits (queue full → overloaded, expired requests, degradation to the
heuristic when busy or when the model is too slow).