Skip to content

Broker Selection Guide

How to choose the right broker for your tasks_worker deployment.

Quick comparison

memory local postgres rabbitmq
Setup None None PostgreSQL RabbitMQ server
Message persistence ❌ RAM ❌ RAM ✅ DB ✅ Queue
Multiple worker nodes ❌ ❌ single-host ✅ ⚠️ partial
Distributed task locks ❌ ❌ ✅ DB ❌ in-memory/node
Distributed worker registry ❌ ❌ ✅ DB ❌ in-memory/node
Cluster-wide concurrency limits ❌ ❌ ✅ ❌ per-node only
Dead-letter queue ✅ ✅ ✅ ✅
Delayed requeue (visibility timeout) ✅ ✅ ✅ available_at ❌ needs delayed-exchange/DLX
Best for Tests / dev Single-host multi-process Production High-throughput / routing

Delayed requeue is nack(msg_id, requeue=True, delay=<seconds>) — the message goes back on the queue invisible until the delay elapses (#20). It matters more than it sounds: without it, "put it back and try later" has no later, because the dispatch loop only pauses when a claim comes back empty. A message requeued and immediately claimable is retried as fast as the broker answers.

nack() returns whether the delay was honoured, and callers must check it. On rabbitmq it returns False and the caller waits in the worker instead — which costs a pool slot, so enough contended messages starve other work (measured: 6 contended messages on 4 workers completed zero other tasks while the same setup with a broker-side delay completed 200). If you rely on per-entity locks under contention, that is a reason to prefer postgres.

nack(..., count_attempt=False) requeues without the redelivery counting as a retry, so lock contention does not inflate attempts until the orphan reaper dead-letters a message that never actually failed.

When to use each broker

memory — in-process, no setup

BROKER_TYPE = "memory"
- Zero dependencies - Messages lost on process restart - Use for: unit tests, local development, single-process apps

local — multi-process on one host

BROKER_TYPE = "local"
BROKER_DSN = "127.0.0.1:50050"   # optional
- Uses multiprocessing.BaseManager for IPC - All workers on the same machine share state via the manager server - Call broker.setup() before starting workers to launch the manager - Use for: staging, local multi-process setups

BROKER_TYPE = "postgres"
BROKER_DSN = "postgresql+psycopg://user:pass@host/db"  # optional; uses FastPluggy core DB if omitted
- Multiple workers on different hosts can compete atomically via SELECT … FOR UPDATE SKIP LOCKED - Worker heartbeats, task locks, and running counters are all DB-persisted - Stale workers auto-cleaned after postgres_worker_ttl_seconds (default: 86400) - Use for: any production deployment; the simplest path to horizontal scaling

rabbitmq — advanced message routing

BROKER_TYPE = "rabbitmq"
BROKER_DSN = "amqp://guest:guest@localhost:5672/"
- AMQP naturally distributes messages across competing consumers - Durable queues survive server restarts - Limitation: worker registry, task locks, and concurrency counters are in-memory per node — not shared across a cluster. Two nodes can exceed per-topic concurrency limits or run the same exclusive task simultaneously. - Use for: deployments that already run RabbitMQ and need advanced exchange/routing features; single-node or when lock correctness is not critical

Choosing between postgres and rabbitmq for multi-node

Question Answer
Do you need cluster-wide exclusive task locks? → postgres
Do you need cluster-wide concurrency limits per topic? → postgres
Do you already run RabbitMQ and need topic-exchange routing? → rabbitmq
Simplest production setup, minimal ops overhead? → postgres (reuses FastPluggy DB)

Configuration reference

Setting Default Description
BROKER_TYPE memory Broker backend: memory, local, postgres, rabbitmq
BROKER_DSN — Connection string (optional for postgres — falls back to core DB)
postgres_worker_ttl_seconds 86400 PostgresBroker: seconds before stale worker records are cleaned up