Add resource limits, abuse protection and overload handling
Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.
- Docker: CPU/memory/pids caps per container (overridable in .env);
uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
rate per connection (token bucket, close 1008 on floods), lobby
creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
throttling, with hysteresis (normal/busy/overloaded, severe lag
overloads immediately)
- Admission control: create_lobby and start_game refused while
overloaded or MAX_ACTIVE_GAMES are running; server_status message
broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
limits; docs updated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,33 @@
|
||||
"""Helpers shared by the server tests."""
|
||||
|
||||
SECRET = "test-secret"
|
||||
|
||||
# Tests play much faster than humans, so the message rate limit is raised.
|
||||
TEST_SETTINGS = dict(
|
||||
ai_shared_secret=SECRET,
|
||||
ai_move_delay=0,
|
||||
trick_pause=0,
|
||||
ai_decision_timeout=2,
|
||||
message_rate=10_000,
|
||||
message_burst=10_000,
|
||||
)
|
||||
|
||||
|
||||
def receive_until(ws, predicate, limit=5000):
|
||||
for _ in range(limit):
|
||||
message = ws.receive_json()
|
||||
if predicate(message):
|
||||
return message
|
||||
raise AssertionError("expected message not received")
|
||||
|
||||
|
||||
def is_state(message):
|
||||
return message["type"] == "state"
|
||||
|
||||
|
||||
def connect(client, token=None):
|
||||
ws = client.sockets.enter_context(client.websocket_connect("/ws"))
|
||||
ws.send_json({"type": "hello", "token": token})
|
||||
welcome = receive_until(ws, lambda m: m["type"] == "welcome")
|
||||
receive_until(ws, is_state)
|
||||
return ws, welcome
|
||||
Reference in New Issue
Block a user