Add resource limits, abuse protection and overload handling
Running games always continue; new work is refused when there is no
capacity; abuse is rejected immediately.
- Docker: CPU/memory/pids caps per container (overridable in .env);
uvicorn message size, queue and concurrency limits
- Server abuse limits: connections total/per IP (close 1013), message
rate per connection (token bucket, close 1008 on floods), lobby
creation rate per IP
- Load monitor: event loop lag, CPU of the own cgroup quota and
throttling, with hysteresis (normal/busy/overloaded, severe lag
overloads immediately)
- Admission control: create_lobby and start_game refused while
overloaded or MAX_ACTIVE_GAMES are running; server_status message
broadcast on changes; /health reports load but stays ok
- AI bridge: timeout sent with requests, in-flight cap, circuit breaker
- AI service: bounded workers and queue ("overloaded"), drops expired
requests, degrades to the heuristic agent when busy or too slow
- Website: busy notices, Create/Start disabled when not accepting
games, slow retry when the server is full
- Tests for limiters, load levels, admission control and AI service
limits; docs updated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,42 +1,9 @@
|
||||
import contextlib
|
||||
import random
|
||||
import threading
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.config import Settings
|
||||
from app.main import create_app
|
||||
|
||||
SECRET = "test-secret"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
settings = Settings(ai_shared_secret=SECRET, ai_move_delay=0, trick_pause=0, ai_decision_timeout=2)
|
||||
with TestClient(create_app(settings)) as test_client, contextlib.ExitStack() as sockets:
|
||||
test_client.sockets = sockets # websocket sessions are closed before the app stops
|
||||
yield test_client
|
||||
|
||||
|
||||
def receive_until(ws, predicate, limit=5000):
|
||||
for _ in range(limit):
|
||||
message = ws.receive_json()
|
||||
if predicate(message):
|
||||
return message
|
||||
raise AssertionError("expected message not received")
|
||||
|
||||
|
||||
def is_state(message):
|
||||
return message["type"] == "state"
|
||||
|
||||
|
||||
def connect(client, token=None):
|
||||
ws = client.sockets.enter_context(client.websocket_connect("/ws"))
|
||||
ws.send_json({"type": "hello", "token": token})
|
||||
welcome = receive_until(ws, lambda m: m["type"] == "welcome")
|
||||
receive_until(ws, is_state)
|
||||
return ws, welcome
|
||||
from helpers import SECRET, connect, is_state, receive_until
|
||||
|
||||
|
||||
def test_health(client):
|
||||
|
||||
Reference in New Issue
Block a user