Add resource limits, abuse protection and overload handling
Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.
- Docker: CPU/memory/pids caps per container (overridable in .env);
uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
rate per connection (token bucket, close 1008 on floods), lobby
creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
throttling, with hysteresis (normal/busy/overloaded, severe lag
overloads immediately)
- Admission control: create_lobby and start_game refused while
overloaded or MAX_ACTIVE_GAMES are running; server_status message
broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
limits; docs updated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -35,5 +35,44 @@ class Settings:
|
||||
)
|
||||
log_level: str = os.environ.get("LOG_LEVEL", "INFO")
|
||||
|
||||
# --- Abuse limits -------------------------------------------------------
|
||||
# Open websocket connections in total and per client IP.
|
||||
max_connections: int = _int("MAX_CONNECTIONS", 500)
|
||||
max_connections_per_ip: int = _int("MAX_CONNECTIONS_PER_IP", 10)
|
||||
# Messages per second a single connection may send (token bucket).
|
||||
message_rate: float = _float("MESSAGE_RATE", 10.0)
|
||||
message_burst: int = _int("MESSAGE_BURST", 30)
|
||||
# Dropped messages after which a flooding connection is closed.
|
||||
max_rate_violations: int = _int("MAX_RATE_VIOLATIONS", 50)
|
||||
# Lobbies a single IP may create per minute.
|
||||
lobby_create_rate: int = _int("LOBBY_CREATE_RATE", 10)
|
||||
|
||||
# --- Load control -------------------------------------------------------
|
||||
# Games running at the same time; further game starts are refused.
|
||||
max_active_games: int = _int("MAX_ACTIVE_GAMES", 50)
|
||||
load_sample_interval: float = _float("LOAD_SAMPLE_INTERVAL", 1.0)
|
||||
# Event loop lag (seconds) / CPU share of the available quota (0..1).
|
||||
load_lag_busy: float = _float("LOAD_LAG_BUSY", 0.1)
|
||||
load_lag_overload: float = _float("LOAD_LAG_OVERLOAD", 0.25)
|
||||
# A single sample with this much lag means overload right away: when the
|
||||
# server is that slow, even the load sampler runs rarely.
|
||||
load_lag_severe: float = _float("LOAD_LAG_SEVERE", 1.0)
|
||||
load_cpu_busy: float = _float("LOAD_CPU_BUSY", 0.7)
|
||||
load_cpu_overload: float = _float("LOAD_CPU_OVERLOAD", 0.9)
|
||||
# Share of CPU periods in which the container was throttled (cgroup v2).
|
||||
load_throttle_overload: float = _float("LOAD_THROTTLE_OVERLOAD", 0.5)
|
||||
# Consecutive overloaded samples to enter / good samples to leave overload.
|
||||
load_overload_samples: int = _int("LOAD_OVERLOAD_SAMPLES", 3)
|
||||
load_recover_samples: int = _int("LOAD_RECOVER_SAMPLES", 10)
|
||||
# "normal", "busy" or "overloaded" to pin the level (testing only).
|
||||
load_force_level: str = os.environ.get("LOAD_FORCE_LEVEL", "")
|
||||
|
||||
# --- AI service protection ----------------------------------------------
|
||||
# Unanswered decide requests per AI connection before falling back.
|
||||
ai_max_inflight: int = _int("AI_MAX_INFLIGHT", 32)
|
||||
# Consecutive AI failures that pause AI requests for ai_breaker_cooldown seconds.
|
||||
ai_breaker_failures: int = _int("AI_BREAKER_FAILURES", 5)
|
||||
ai_breaker_cooldown: float = _float("AI_BREAKER_COOLDOWN", 30.0)
|
||||
|
||||
|
||||
settings = Settings()
|
||||
|
||||
Reference in New Issue
Block a user