* fix(agents): recognize connection errors as retryable timeout failures
## Problem
When a model endpoint becomes unreachable (e.g., local proxy down,
relay server offline), the failover system fails to switch to the
next candidate model. Errors like "Connection error." are not
classified as retryable, causing the session to hang on a broken
endpoint instead of falling back to healthy alternatives.
## Root Cause
Connection/network errors are not recognized by the current failover
classifier:
- Text patterns like "Connection error.", "fetch failed", "network error"
- Error codes like ECONNREFUSED, ENOTFOUND, EAI_AGAIN (in message text)
While `failover-error.ts` handles these as error codes (err.code),
it misses them when they appear as plain text in error messages.
## Solution
Extend timeout error patterns to include connection/network failures:
**In `errors.ts` (ERROR_PATTERNS.timeout):**
- Text: "connection error", "network error", "fetch failed", etc.
- Regex: /\beconn(?:refused|reset|aborted)\b/i, /\benotfound\b/i, /\beai_again\b/i
**In `failover-error.ts` (TIMEOUT_HINT_RE):**
- Same patterns for non-assistant error paths
## Testing
Added test cases covering:
- "Connection error."
- "fetch failed"
- "network error: ECONNREFUSED"
- "ENOTFOUND" / "EAI_AGAIN" in message text
## Impact
- **Compatibility:** High - only expands retryable error detection
- **Behavior:** Connection failures now trigger automatic fallback
- **Risk:** Low - changes are additive and well-tested
* style: fix code formatting for test file
Classify Anthropic's 529 status code as "rate_limit" so model fallback
triggers reliably without depending on fragile message-based detection.
Closes#28502
Previously, only ETIMEDOUT / ESOCKETTIMEDOUT / ECONNRESET / ECONNABORTED
were recognised as failover-worthy network errors. Connection-level
failures such as ECONNREFUSED (server down), ENETUNREACH / EHOSTUNREACH
(network disconnected), ENETRESET, and EAI_AGAIN (DNS failure) were
treated as unknown errors and did not advance the fallback chain.
This is particularly impactful when a local fallback model (e.g. Ollama)
is configured: if the remote provider is unreachable due to a network
outage, the gateway should fall back to the local model instead of
returning an error to the user.
Add the missing error codes to resolveFailoverReasonFromError() and
corresponding e2e tests.
Closes#18868
* fix: handle CLI session expired errors gracefully
- Add session_expired to FailoverReason type
- Add isCliSessionExpiredErrorMessage to detect expired CLI sessions
- Modify runCliAgent to retry with new session when session expires
- Update agentCommand to clear expired session IDs from session store
- Add proper error handling to prevent gateway crashes on expired sessions
Fixes#30986
* fix: add session_expired to AuthProfileFailureReason and missing log import
* fix: type cli-runner usage field to match EmbeddedPiAgentMeta
* fix: harden CLI session-expiry recovery handling
* build: regenerate host env security policy swift
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* fix: treat HTTP 502/503/504 as failover-eligible (timeout reason)
When a model API returns 502 Bad Gateway, 503 Service Unavailable, or
504 Gateway Timeout, the error object carries the status code directly.
resolveFailoverReasonFromError() only checked 402/429/401/403/408/400,
so 5xx server errors fell through to message-based classification which
requires the status code to appear at the start of the error message.
Many API SDKs (Google, Anthropic) set err.status = 503 without prefixing
the message with '503', so the message classifier never matched and
failover never triggered — the run retried the same broken model.
Add 502/503/504 to the status-code branch, returning 'timeout' (matching
the existing behavior of isTransientHttpError in the message classifier).
Fixes#20999
* Changelog: add failover 502/503/504 note with credits
* Failover: classify HTTP 504 as transient in message parser
* Changelog: credit taw0002 and vincentkoc for failover fix
---------
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
* fix: treat HTTP 503 as failover-eligible for LLM provider errors
When LLM SDKs wrap 503 responses, the leading "503" prefix is lost
(e.g. Google Gemini returns "high demand" / "UNAVAILABLE" without a
numeric prefix). The existing isTransientHttpError only matches
messages starting with "503 ...", so these wrapped errors silently
skip failover — no profile rotation, no model fallback.
This patch closes that gap:
- resolveFailoverReasonFromError: map HTTP status 503 → rate_limit
(covers structured error objects with a status field)
- ERROR_PATTERNS.overloaded: add /\b503\b/, "service unavailable",
"high demand" (covers message-only classification when the leading
status prefix is absent)
Existing isTransientHttpError behavior is unchanged; these additions
are complementary and only fire for errors that previously fell
through unclassified.
* fix: address review feedback — drop /\b503\b/ pattern, add test coverage
- Remove `/\b503\b/` from ERROR_PATTERNS.overloaded to resolve the
semantic inconsistency noted by reviewers: `isTransientHttpError`
already handles messages prefixed with "503" (→ "timeout"), so a
redundant overloaded pattern would classify the same class of errors
differently depending on message formatting.
- Keep "service unavailable" and "high demand" patterns — these are the
real gap-fillers for SDK-rewritten messages that lack a numeric prefix.
- Add test case for JSON-wrapped 503 error body containing "overloaded"
to strengthen coverage.
* fix: unify 503 classification — status 503 → timeout (consistent with isTransientHttpError)
resolveFailoverReasonFromError previously mapped status 503 → "rate_limit",
while the string-based isTransientHttpError mapped "503 ..." → "timeout".
Align both paths: structured {status: 503} now also returns "timeout",
matching the existing transient-error convention. Both reasons are
failover-eligible, so runtime behavior is unchanged.
---------
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
When streaming providers (GLM, OpenRouter, etc.) return 'stop reason: abort'
due to stream interruption, OpenClaw's failover mechanism did not recognize
this as a timeout condition. This prevented fallback models from being
triggered, leaving users with failed requests instead of graceful failover.
Changes:
- Add abort patterns to ERROR_PATTERNS.timeout in pi-embedded-helpers/errors.ts
- Extend TIMEOUT_HINT_RE regex to include abort patterns in failover-error.ts
Fixes#18453
Co-authored-by: James <james@openclaw.ai>
