Don't trip the AI circuit breaker on load shedding
With many simultaneous games, AI turns start in bursts and exceed the in-flight cap. That was counted as an AI failure, so after 5 bursts the breaker paused the AI for 30 s and every AI seat played random moves. - Only real failures (timeouts, lost connection, malformed/illegal answers, unexpected errors) count towards the breaker; the in-flight cap and "overloaded"/"expired" answers from the AI service do not - AI_MAX_INFLIGHT default 32 -> 64 (>= MAX_ACTIVE_GAMES, matches the AI service's AI_MAX_QUEUE) - Test that load shedding keeps the breaker closed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -15,9 +15,15 @@ requests are distributed round robin.
|
||||
A random legal action is played instead (so a game never gets stuck) when no
|
||||
instance is connected, it does not answer in time, it answers with an error or
|
||||
an illegal action, too many requests are already waiting for it
|
||||
(``max_inflight``), or the circuit breaker is open: after ``breaker_failures``
|
||||
failures in a row the AI is not asked for ``breaker_cooldown`` seconds, so a
|
||||
struggling AI service is not buried under even more requests.
|
||||
(``max_inflight``), or the circuit breaker is open.
|
||||
|
||||
The circuit breaker protects against a *broken or hanging* AI service: after
|
||||
``breaker_failures`` real failures in a row (timeouts, lost connection,
|
||||
malformed or illegal answers, unexpected errors) the AI is not asked for
|
||||
``breaker_cooldown`` seconds, so games don't wait ``timeout`` seconds for every
|
||||
AI move. Deliberate load shedding is *not* a failure and does not trip it: the
|
||||
in-flight cap, and the AI answering "overloaded" or "expired" (it is healthy
|
||||
and answered immediately).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -36,6 +42,9 @@ from wizard_game import Action, Observation, action_from_dict
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Errors with which a healthy AI service sheds load (see wizard_ai/service/client.py).
|
||||
LOAD_SHEDDING_ERRORS = ("overloaded", "expired")
|
||||
|
||||
|
||||
class AIConnection:
|
||||
def __init__(self, websocket: WebSocket, name: str) -> None:
|
||||
@@ -94,7 +103,7 @@ class AIBridge:
|
||||
if loop.time() < self._breaker_open_until:
|
||||
return self._fallback(observation, "AI paused after repeated failures", logging.DEBUG)
|
||||
if len(connection.pending) >= self._max_inflight:
|
||||
return self._failure(observation, f"{len(connection.pending)} AI requests already waiting")
|
||||
return self._fallback(observation, f"{len(connection.pending)} AI requests already waiting")
|
||||
|
||||
request_id = uuid.uuid4().hex
|
||||
future: asyncio.Future[dict[str, Any]] = loop.create_future()
|
||||
@@ -113,7 +122,11 @@ class AIBridge:
|
||||
connection.pending.pop(request_id, None)
|
||||
|
||||
if reply.get("type") != "action":
|
||||
return self._failure(observation, f"AI error: {reply.get('message')}")
|
||||
message = reply.get("message")
|
||||
if message in LOAD_SHEDDING_ERRORS:
|
||||
self._consecutive_failures = 0 # the service is alive and protecting itself
|
||||
return self._fallback(observation, f"AI service {message}")
|
||||
return self._failure(observation, f"AI error: {message}")
|
||||
try:
|
||||
action = action_from_dict(reply.get("action", {}))
|
||||
except ValueError as exc:
|
||||
|
||||
Reference in New Issue
Block a user