With many simultaneous games, AI turns start in bursts and exceed the
in-flight cap. That was counted as an AI failure, so after 5 bursts the
breaker paused the AI for 30 s and every AI seat played random moves.
- Only real failures (timeouts, lost connection, malformed/illegal
answers, unexpected errors) count towards the breaker; the in-flight
cap and "overloaded"/"expired" answers from the AI service do not
- AI_MAX_INFLIGHT default 32 -> 64 (>= MAX_ACTIVE_GAMES, matches the AI
service's AI_MAX_QUEUE)
- Test that load shedding keeps the breaker closed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.
- Docker: CPU/memory/pids caps per container (overridable in .env);
uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
rate per connection (token bucket, close 1008 on floods), lobby
creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
throttling, with hysteresis (normal/busy/overloaded, severe lag
overloads immediately)
- Admission control: create_lobby and start_game refused while
overloaded or MAX_ACTIVE_GAMES are running; server_status message
broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
limits; docs updated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
marvinandClaude Opus 5.5