Add resource limits, abuse protection and overload handling

Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.

- Docker: CPU/memory/pids caps per container (overridable in .env);
  uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
  rate per connection (token bucket, close 1008 on floods), lobby
  creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
  throttling, with hysteresis (normal/busy/overloaded, severe lag
  overloads immediately)
- Admission control: create_lobby and start_game refused while
  overloaded or MAX_ACTIVE_GAMES are running; server_status message
  broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
  requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
  games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
  limits; docs updated

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-27 12:51:46 +02:00
co-authored by Claude Opus 5.5
parent 905574e8f1
commit 87dfcd55b7
33 changed files with 1591 additions and 128 deletions
+39
View File
@@ -35,5 +35,44 @@ class Settings:
)
log_level: str = os.environ.get("LOG_LEVEL", "INFO")
# --- Abuse limits -------------------------------------------------------
# Open websocket connections in total and per client IP.
max_connections: int = _int("MAX_CONNECTIONS", 500)
max_connections_per_ip: int = _int("MAX_CONNECTIONS_PER_IP", 10)
# Messages per second a single connection may send (token bucket).
message_rate: float = _float("MESSAGE_RATE", 10.0)
message_burst: int = _int("MESSAGE_BURST", 30)
# Dropped messages after which a flooding connection is closed.
max_rate_violations: int = _int("MAX_RATE_VIOLATIONS", 50)
# Lobbies a single IP may create per minute.
lobby_create_rate: int = _int("LOBBY_CREATE_RATE", 10)
# --- Load control -------------------------------------------------------
# Games running at the same time; further game starts are refused.
max_active_games: int = _int("MAX_ACTIVE_GAMES", 50)
load_sample_interval: float = _float("LOAD_SAMPLE_INTERVAL", 1.0)
# Event loop lag (seconds) / CPU share of the available quota (0..1).
load_lag_busy: float = _float("LOAD_LAG_BUSY", 0.1)
load_lag_overload: float = _float("LOAD_LAG_OVERLOAD", 0.25)
# A single sample with this much lag means overload right away: when the
# server is that slow, even the load sampler runs rarely.
load_lag_severe: float = _float("LOAD_LAG_SEVERE", 1.0)
load_cpu_busy: float = _float("LOAD_CPU_BUSY", 0.7)
load_cpu_overload: float = _float("LOAD_CPU_OVERLOAD", 0.9)
# Share of CPU periods in which the container was throttled (cgroup v2).
load_throttle_overload: float = _float("LOAD_THROTTLE_OVERLOAD", 0.5)
# Consecutive overloaded samples to enter / good samples to leave overload.
load_overload_samples: int = _int("LOAD_OVERLOAD_SAMPLES", 3)
load_recover_samples: int = _int("LOAD_RECOVER_SAMPLES", 10)
# "normal", "busy" or "overloaded" to pin the level (testing only).
load_force_level: str = os.environ.get("LOAD_FORCE_LEVEL", "")
# --- AI service protection ----------------------------------------------
# Unanswered decide requests per AI connection before falling back.
ai_max_inflight: int = _int("AI_MAX_INFLIGHT", 32)
# Consecutive AI failures that pause AI requests for ai_breaker_cooldown seconds.
ai_breaker_failures: int = _int("AI_BREAKER_FAILURES", 5)
ai_breaker_cooldown: float = _float("AI_BREAKER_COOLDOWN", 30.0)
settings = Settings()