Add resource limits, abuse protection and overload handling

Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.

- Docker: CPU/memory/pids caps per container (overridable in .env);
  uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
  rate per connection (token bucket, close 1008 on floods), lobby
  creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
  throttling, with hysteresis (normal/busy/overloaded, severe lag
  overloads immediately)
- Admission control: create_lobby and start_game refused while
  overloaded or MAX_ACTIVE_GAMES are running; server_status message
  broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
  requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
  games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
  limits; docs updated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-27 12:51:46 +02:00
co-authored by Claude Opus 5.5
parent 905574e8f1
commit 87dfcd55b7
33 changed files with 1591 additions and 128 deletions
+22
View File
@@ -9,3 +9,25 @@ ALLOWED_ORIGINS=
AI_AGENT=model
LOG_LEVEL=INFO
# --- Resource limits (hard caps per container) -------------------------------
SERVER_CPUS=1.0
SERVER_MEMORY=512M
AI_CPUS=1.0
AI_MEMORY=1G
WEBSITE_CPUS=0.25
WEBSITE_MEMORY=64M
# --- Capacity -----------------------------------------------------------------
# Games running at the same time; more game starts are refused ("server busy").
MAX_ACTIVE_GAMES=50
# Open websocket connections in total / per client IP.
MAX_CONNECTIONS=500
MAX_CONNECTIONS_PER_IP=10
# --- AI service ---------------------------------------------------------------
# Decisions computed in parallel, waiting requests before refusing, and queue
# depth from which the cheap heuristic agent is used instead of the model.
AI_WORKERS=1
AI_MAX_QUEUE=64
AI_DEGRADE_QUEUE=8