feat(jobs): Redis-backed job queue as alternative to PostgreSQL polling #103

New Issue

forbes · 2026-02-14T20:06:42Z

forbes commented

2026-02-14 20:06:42 +00:00

Ref: docs/WORKERS.md, docs/MODULES.md §3.9

The current job queue uses SELECT FOR UPDATE SKIP LOCKED against PostgreSQL for atomic job claims. This works well at low volume but adds polling load to the database as runner count grows.

Add Redis as an optional, higher-throughput job queue backend:

Scope:

Redis list (BRPOPLPUSH / BLMOVE) for job dispatch — runners block-wait instead of polling
Redis pub/sub for real-time job lifecycle events (claimed, progress, completed, failed)
PostgreSQL remains the durable store for job state, logs, and history
Redis is the fast path: enqueue into Redis, runner pops from Redis, writes results back to PostgreSQL
Graceful fallback: if Redis is unavailable, fall back to PostgreSQL polling (current behavior)

Config:

jobs:
  queue_backend: redis  # or "postgres" (default)
  redis:
    url: redis://localhost:6379/0
    prefix: silo:jobs:

Key design points:

Jobs table remains the source of truth — Redis is a delivery optimization
Runner claim semantics unchanged from the caller's perspective
Job timeout and stale runner expiry still handled by PostgreSQL sweepers
Redis pub/sub can feed SSE events for lower-latency UI updates

Not in scope:

Redis for session storage (separate concern)
Redis for caching (future consideration)

Depends on #98 (module system — jobs module gate).

**Ref:** docs/WORKERS.md, docs/MODULES.md §3.9 The current job queue uses `SELECT FOR UPDATE SKIP LOCKED` against PostgreSQL for atomic job claims. This works well at low volume but adds polling load to the database as runner count grows. Add Redis as an optional, higher-throughput job queue backend: **Scope:** - Redis list (`BRPOPLPUSH` / `BLMOVE`) for job dispatch — runners block-wait instead of polling - Redis pub/sub for real-time job lifecycle events (claimed, progress, completed, failed) - PostgreSQL remains the durable store for job state, logs, and history - Redis is the fast path: enqueue into Redis, runner pops from Redis, writes results back to PostgreSQL - Graceful fallback: if Redis is unavailable, fall back to PostgreSQL polling (current behavior) **Config:** ```yaml jobs: queue_backend: redis # or "postgres" (default) redis: url: redis://localhost:6379/0 prefix: silo:jobs: ``` **Key design points:** - Jobs table remains the source of truth — Redis is a delivery optimization - Runner claim semantics unchanged from the caller's perspective - Job timeout and stale runner expiry still handled by PostgreSQL sweepers - Redis pub/sub can feed SSE events for lower-latency UI updates **Not in scope:** - Redis for session storage (separate concern) - Redis for caching (future consideration) Depends on #98 (module system — jobs module gate).

forbes referenced this issue

2026-02-15 10:54:36 +00:00

feat(jobs): wire auto-triggering on revision_created and bom_changed events #107

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kindred/silo#103