Skip to content

Broker Selection Guide

How to choose the right broker for your tasks_worker deployment.

Quick comparison

memory local postgres rabbitmq
Setup None None PostgreSQL RabbitMQ server
Message persistence ❌ RAM ❌ RAM ✅ DB ✅ Queue
Multiple worker nodes ❌ ❌ single-host ✅ ⚠️ partial
Distributed task locks ❌ ❌ ✅ DB ⚠️ token queue, unsafe past 300 s TTL
Distributed worker registry ❌ ❌ ✅ DB ⚠️ per-node view built from broadcasts
Cluster-wide concurrency limits ❌ ❌ ✅ ❌ per-node only
Dead-letter queue ✅ ✅ ✅ ✅
Delayed requeue (visibility timeout) ✅ ✅ ✅ available_at ❌ needs delayed-exchange/DLX
Best for Tests / dev Single-host multi-process Production High-throughput / routing

Delayed requeue is nack(msg_id, requeue=True, delay=<seconds>) — the message goes back on the queue invisible until the delay elapses (#20). It matters more than it sounds: without it, "put it back and try later" has no later, because the dispatch loop only pauses when a claim comes back empty. A message requeued and immediately claimable is retried as fast as the broker answers.

nack() returns whether the delay was honoured, and callers must check it. On rabbitmq it returns False and the caller waits in the worker instead — which costs a pool slot, so enough contended messages starve other work (measured: 6 contended messages on 4 workers completed zero other tasks while the same setup with a broker-side delay completed 200). If you rely on per-entity locks under contention, that is a reason to prefer postgres.

nack(..., count_attempt=False) requeues without the redelivery counting as a retry, so lock contention does not inflate attempts until the orphan reaper dead-letters a message that never actually failed.

When to use each broker

memory — in-process, no setup

BROKER_TYPE = "memory"
- Zero dependencies - Messages lost on process restart - Use for: unit tests, local development, single-process apps

local — multi-process on one host

BROKER_TYPE = "local"
BROKER_DSN = "127.0.0.1:50050"   # optional
- Uses multiprocessing.BaseManager for IPC - All workers on the same machine share state via the manager server - The first process to call setup() launches the manager; the others reuse it (see technical.md) - Use for: staging, local multi-process setups

BROKER_TYPE = "postgres"
BROKER_DSN = "postgresql+psycopg://user:pass@host/db"  # optional; uses FastPluggy core DB if omitted
- Multiple workers on different hosts can compete atomically via SELECT … FOR UPDATE SKIP LOCKED - Worker heartbeats, task locks, and running counters are all DB-persisted - Stale workers auto-cleaned after postgres_worker_ttl_seconds (default: 86400) - Use for: any production deployment; the simplest path to horizontal scaling

rabbitmq — advanced message routing

BROKER_TYPE = "rabbitmq"
BROKER_DSN = "amqp://guest:guest@localhost:5672/"
- AMQP naturally distributes messages across competing consumers - Durable queues survive server restarts - Limitation: concurrency counters are per node. N nodes can run up to N × a topic's limit. - Limitation: the worker registry is a per-node view rebuilt from fp.workers broadcasts, so a node that missed them sees an incomplete cluster until the next heartbeats. - Only nodes that need the cluster view subscribe to fp.workers (#43). Every subscriber gets a copy of every heartbeat/lifecycle/lock/counter broadcast (≈12k msgs/h each), plus a second connection and a consumer thread. Subscribing happens automatically on register_worker() (workers, beat), in the plugin's web-UI listener, and on the first read of cluster state (get_workers(), get_cluster_running_tasks(), acquire_lock()). A process that only setup()s and submit()s tasks stays publish-only, and its broadcasts (e.g. topic config) are still published to the fanout. Call broker.enable_cluster_view() to opt in explicitly. A node that subscribes late does not see events broadcast before it subscribed. - Limitation: locks are a token in a tw.lock.<name> queue with a 300 s TTL that is never renewed. Past the TTL, the holder's own maintenance releases the token and another node can re-seed it, so a task running longer than 300 s can run on two nodes at once. Verified by a probe in #31. - Use for: deployments that already run RabbitMQ and need advanced exchange/routing features, where exclusive tasks stay well under 300 s or lock correctness is not critical

Choosing between postgres and rabbitmq for multi-node

Question Answer
Do you need cluster-wide exclusive task locks? → postgres
Do you need cluster-wide concurrency limits per topic? → postgres
Do you already run RabbitMQ and need topic-exchange routing? → rabbitmq
Simplest production setup, minimal ops overhead? → postgres (reuses FastPluggy DB)

Configuration reference

Setting Default Description
BROKER_TYPE local Broker backend: memory, local, postgres, rabbitmq (none disables the broker)
BROKER_DSN — Connection string (optional for postgres — falls back to core DB)
postgres_worker_ttl_seconds 86400 PostgresBroker: seconds before stale worker records are cleaned up