SubLaneSubLane

Workspace alerts

Optional notifications for account authorization, pool availability, and generation failures.

Workspace administrators can configure alerts from System settings → Workspace alerts. Notifications are off until an administrator saves a destination and enables them. Each workspace has its own destination and incident history.

Use a public HTTPS endpoint accepting JSON POST requests. Private/local addresses, redirects, embedded usernames/passwords, and URL fragments are rejected. SubLane encrypts the destination, displays only its host, and never automatically reveals it. Leave the URL blank when editing to keep it. Removing the destination disables notifications and deletes its pending notification history.

Conditions

EventCondition
account_reauthorizationAn enabled subscription account requires authorization again.
pool_unavailableAn enabled pool has no enabled, verified subscription account outside recorded cooldown/recovery. Empty pools also match.
request_failuresIn the last five minutes, at least five generations finished and at least half ended in an error or incomplete response. Canceled and locally rejected requests are excluded.

Checks use local persisted state approximately once per minute; they do not send model requests or test subscription credentials. The original pool alert checks account health. Optionally enable model alerts and configure a used-quota threshold from 1–99 (blank disables it); both are off by default. Only permitted, previously observed models and fresh quotas are evaluated. Unknown or stale observations cannot prove failure or recovery; concurrency saturation alone does not trigger a model alert. Idle snapshots can expire, and checks do not refresh metadata. A cleared cooldown after a successful request or an administrator's recovery action can resolve a pool incident.

Delivery and recovery

An unchanged incident sends one notification. A condition that clears after its firing event was accepted sends a resolved event. Incident state survives restart. Delivery failures retry after one minute, then back off up to 30 minutes; each pass attempts at most ten deliveries across the instance. Large pending queues may take additional passes.

The receiver must return a 2xx status within five seconds. Deduplicate using id, which is also sent in the Idempotency-Key header: a lost acknowledgment or a restart between receiver acceptance and local persistence can cause a repeated delivery. Saving settings sends no test. After saving a destination, explicitly Send test notification, even with notifications off. Its kind and status are test. One test per minute is permitted across restarts; failures are not retried and incident state is unchanged. The settings page displays delivery failure and the most recent accepted notification time; saving a URL alone does not prove it works.

Example event using synthetic IDs:

{
  "id": "11111111111111111111111111111111",
  "workspace_id": 1,
  "kind": "pool_unavailable",
  "subject": "2",
  "status": "firing",
  "observed_at": 1900000000
}

Model events use model_unavailable, with pool ID and native model ID joined by a colon as their subject. Quota events use quota_low, with an account ID as their subject.

Only identifiers (including the native model ID for model events), condition type, state, and observation time are sent. Account names, member identities, keys, webhook URLs, prompts, responses, and raw errors are excluded. This is a generic webhook format; chat applications may require an adapter to their message format.

Backups include encrypted destinations and incident history with the matching vault key. Restoring an older snapshot can replay subsequent incidents, so receivers should keep their deduplication history.