Add resource limits, abuse protection and overload handling

Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.

- Docker: CPU/memory/pids caps per container (overridable in .env);
  uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
  rate per connection (token bucket, close 1008 on floods), lobby
  creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
  throttling, with hysteresis (normal/busy/overloaded, severe lag
  overloads immediately)
- Admission control: create_lobby and start_game refused while
  overloaded or MAX_ACTIVE_GAMES are running; server_status message
  broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
  requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
  games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
  limits; docs updated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-27 12:51:46 +02:00
co-authored by Claude Opus 5.5
parent 905574e8f1
commit 87dfcd55b7
33 changed files with 1591 additions and 128 deletions
+28
View File
@@ -4,6 +4,12 @@ services:
wizard_website:
build: ./wizard_website
restart: unless-stopped
deploy:
resources:
limits:
cpus: "${WEBSITE_CPUS:-0.25}"
memory: ${WEBSITE_MEMORY:-64M}
pids_limit: 64
networks:
- wizard_website
@@ -14,6 +20,16 @@ services:
AI_SHARED_SECRET: ${AI_SHARED_SECRET:?set AI_SHARED_SECRET in .env}
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
MAX_ACTIVE_GAMES: ${MAX_ACTIVE_GAMES:-50}
MAX_CONNECTIONS: ${MAX_CONNECTIONS:-500}
MAX_CONNECTIONS_PER_IP: ${MAX_CONNECTIONS_PER_IP:-10}
# Hard caps so Wizard can never starve other services on the host.
deploy:
resources:
limits:
cpus: "${SERVER_CPUS:-1.0}"
memory: ${SERVER_MEMORY:-512M}
pids_limit: 256
networks:
- wizard_api
- wizard_internal
@@ -31,6 +47,18 @@ services:
AGENT: ${AI_AGENT:-model}
MODEL_PATH: /srv/models/latest
LOG_LEVEL: ${LOG_LEVEL:-INFO}
AI_WORKERS: ${AI_WORKERS:-1}
AI_MAX_QUEUE: ${AI_MAX_QUEUE:-64}
AI_DEGRADE_QUEUE: ${AI_DEGRADE_QUEUE:-8}
# Numeric libraries (numpy/torch) must not start one thread per host core.
OMP_NUM_THREADS: ${AI_WORKERS:-1}
MKL_NUM_THREADS: ${AI_WORKERS:-1}
deploy:
resources:
limits:
cpus: "${AI_CPUS:-1.0}"
memory: ${AI_MEMORY:-1G}
pids_limit: 256
volumes:
# Trained checkpoints; replace without rebuilding the image.
- ./wizard_ai/models:/srv/models:ro