Fallback chains + timeouts

FallbackModel wraps an ordered list of models and serves from the first one that can. When the current model fails with a fallback-worthy error, the chain advances to the next model; the last model’s errors always propagate — there is nothing left to fall to.

from loomflow import Agent
from loomflow.model import AnthropicModel, OpenAIModel, FallbackModel
from loomflow.model.retrying import RetryingModel
from loomflow.governance import RetryPolicy
 
model = FallbackModel([
    RetryingModel(AnthropicModel("claude-opus-4-7",
                                 request_timeout_s=120), RetryPolicy()),
    RetryingModel(OpenAIModel("gpt-4o",
                              request_timeout_s=120), RetryPolicy()),
])
 
agent = Agent("You are a support assistant.", model=model)

A provider outage on the primary now degrades to the secondary instead of failing the run.

Composition order is part of the contract

Put RetryingModel inside the chain, one per member:

FallbackModel([
    RetryingModel(primary,   RetryPolicy()),   # retries exhaust here first
    RetryingModel(secondary, RetryPolicy()),   # then this gets its own budget
])

RetryingModel only raises its final TransientModelError after its policy is spent — so retries exhaust on the primary first, and only then does FallbackModel advance to the secondary, which gets a fresh retry budget.

Wrapping the other way round — RetryingModel(FallbackModel([...])) — would re-run the whole chain on every retry attempt: the secondary gets hammered while the primary’s rate limit is still cooling down. See RetryPolicy + error taxonomy for what each retry attempt does.

default_fall_on — when the chain advances

The fall_on predicate decides, per exception, whether to advance. The default (default_fall_on) fails over on:

• TransientModelError (including RateLimitError) — rate limits, 5xx / overloaded, network blips, per-request timeouts. When the member is a RetryingModel these only surface after its retry budget is exhausted, so advancing is the correct next escalation.
• Plain PermanentModelError — provider statuses the classifier couldn’t pin to a subclass (odd 4xx/5xx); a different provider may well not share them.

It never fails over on:

• AuthenticationError / InvalidRequestError — caller or config problems that another model won’t fix; surface them.
• ContentFilterError — silently rerouting a content-filter rejection to a different provider would be a policy bypass, not resilience.
• Unclassified exceptions — programming errors should surface, not be papered over by a model switch.

Pass your own predicate for different rules:

model = FallbackModel(
    [primary, secondary],
    fall_on=lambda exc: isinstance(exc, RateLimitError),  # rate limits only
)

Streaming: fail over only before the first chunk

Streaming failover mirrors RetryingModel’s discipline: the chain may advance only while waiting for the first chunk. Once a chunk has been yielded to the consumer the stream is committed to that model — a mid-stream switch would duplicate or drop content the consumer already saw, so mid-stream errors propagate unchanged. No silent model switch, no duplicated tokens.

Attribution and capability flags

• last_served records the name of the model that served the most recent successful call, so telemetry can see when the chain failed over. Cost is already honest — the adapter that actually served prices its own usage in usage.cost_usd.
• The wrapper’s stable name is the primary’s (audit / telemetry consistency across calls).
• Capability flags (supports_native_structured_output, count_tokens, …) are delegated to the primary — they are read before the framework knows which member will serve.

result = await agent.run("...")
print(model.last_served)     # "gpt-4o" → the chain failed over

request_timeout_s= — bound every request

All three network adapters — AnthropicModel, OpenAIModel, LiteLLMModel — accept request_timeout_s (default None = unbounded):

AnthropicModel("claude-opus-4-7", request_timeout_s=120)
OpenAIModel("gpt-4o", request_timeout_s=90)
LiteLLMModel("mistral/mistral-large-latest", request_timeout_s=90)

Enforcement is belt-and-suspenders:

1. The value is forwarded as the SDK’s per-request timeout= option, so the HTTP layer applies its own connect/read timeouts.
2. The call is bounded by an anyio wall clock. Non-streaming calls run inside one fail_after; streaming iteration charges every await of the next SSE event against one shared absolute deadline — a hung stream (socket open, no events arriving) is killed the moment the deadline passes.

Timeouts surface as TransientModelError, so RetryingModel retries them and FallbackModel fails over on them. Mid-stream, the usual streaming discipline applies: chunks already yielded are never replayed, so the error propagates to the consumer.

Recommended production stack

model = FallbackModel([
    RetryingModel(
        AnthropicModel("claude-opus-4-7", request_timeout_s=120),
        RetryPolicy(),                      # 3 attempts, exp backoff
    ),
    RetryingModel(
        OpenAIModel("gpt-4o", request_timeout_s=120),
        RetryPolicy(max_attempts=2),        # lighter budget on the backup
    ),
])

Timeouts bound each request → retries absorb transient blips → fallback absorbs sustained provider outages. Each layer escalates to the next only when it has genuinely given up.