Queues and workers

Six BullMQ queues carry every piece of work that must not happen inside a request. All six share BullmqJobQueue, a queue+worker pair over one Redis connection: enqueueing works from construction, but the worker only starts in start(), wired to onModuleInit so no job is consumed before the module graph is fully initialised.

The queues

Concurrency for five of them comes from INBOX_WEBHOOK_CONCURRENCY (default 5). publish-dispatch sets attempts: 3 with exponential backoff from 5s, safe because dispatchGroup is idempotent through its send-state machine.

One job per campaign, not per recipient

campaign-send runs the whole recipient loop in a single job. Fanning out per recipient was rejected on the real numbers: the largest campaign in the estate is 12,198 recipients, and per-recipient jobs would mean 12,198 enqueues, job records, completions and lock renewals for a workload that is not bounded by the worker at all. It is bounded by the per-connection rate limiter, which already serialises every send on that connection — so 12,198 workers would queue behind one token bucket and convert a paced blast into 12,198 rate-limit rejections.

The two things per-recipient jobs usually buy are already provided. Retry: a per-recipient failure is caught inside the loop and recorded on that recipient's row (status: 'failed', errorMessage), and the loop continues. Progress: the recipient rows are the progress bar, and GET /:id/recipients?status=sent is live while the job runs.

Crash-safety rests on the loop only ever attempting recipients whose status is pending or queued. A worker that dies at recipient 7,000 leaves 7,000 committed settled rows; the retry re-reads them and resumes at 7,001. The re-run is a resume, not a replay.

flow-tick is a self-re-enqueueing job

There is no cron. The tick does a sweep and then enqueues its own successor with a delay, which beats a cron entry on three counts: the delayed job lives in Redis and survives a restart with no second moving part; it cannot overlap itself, because the successor is enqueued only after the sweep finishes; and it is drainable in tests.

Without Redis

When INBOX_REDIS_URL is absent every queue is replaced by an in-memory twin — InMemoryWebhookQueue, InMemoryCampaignQueue, and so on. They exist so dev and tests run with no Redis, and they are faithful enough to develop against.

They are not a production fallback. Two consequences follow directly from the jobs living in process memory:

1. Nothing survives a restart. A scheduled campaign, a delayed retry, a queued import — all gone on deploy, with no dead-letter and no trace.
2. Nothing fans out across instances. Each process holds its own queue, so a job enqueued by one is invisible to the others.

The twins also differ in when they run a job. InMemoryPublishQueue and InMemoryOutboundQueue invoke the handler inline the moment enqueue is called. InMemoryCampaignQueue and InMemoryFlowQueue deliberately do not — they hold jobs in a waiting list drained explicitly, because a campaign that settles every recipient before the 202 is written erases the window in which a cancel-after-dispatch or an unsubscribe-after-dispatch is meaningful.

Check which mode you are in with /v1/health: backends.redis false means the twins.

Background work that is not a queue

CampaignSchedulerService sweeps every 60 seconds for campaigns whose scheduledAt has passed and promotes them onto campaign-send. It re-derives due-ness from the database row each tick rather than trusting a delayed job, so a restart, a reschedule and a cancel all behave correctly; a guard flag stops a slow sweep overlapping the next tick. It skips itself under INBOX_ENV=test, where specs drive runDueSweep directly.

Before it existed, create() gave a scheduled campaign the status scheduled and nothing ever dispatched it — a "send later" campaign was accepted, shown as scheduled, and never sent.

InboxBackfillService replays a newly-connected account's history into threads and entries. It is fire-and-forget and never blocks or fails the connect that triggered it, and every ingest runs with silent: true — no realtime events, no outbound webhooks, no auto-responses. Historical messages must not fire the side effects a live message does, or connecting an account would blast every contact with an out-of-office reply. Re-running is a no-op through the @@unique(threadId, externalEntryId) dedup.

Diagnosing a stuck queue

Work through it in this order — the first three account for most cases.

1. Is Redis wired? curl -s $INBOX/v1/health and read backends.redis. False means in-memory twins, and "the job vanished on deploy" is fully explained.
2. Is a worker running? Enqueueing works from construction, but consumption needs start(). Look for the queue's boot log; a process that enqueues fine and consumes nothing is a worker that never started.
3. Did the enqueue fail silently? The queues differ deliberately. webhook-retry and inbox-import reject on a failed enqueue, so the caller surfaces it — a webhook intake failure reaches the sender, which retries. outbound-retry, publish-dispatch and campaign-send are best-effort: the row is already persisted and recoverable, so a failed enqueue is logged, not thrown. Grep the logs for Failed to enqueue.
4. Check the failure log. Every worker logs Job <id> failed: <message> under BullmqJobQueue:<name>.
5. Check the domain rows, not the queue. Campaign recipients, PostDispatch rows and ImportJob.status each record their own progress, and are the truthful answer to "how far did it get". A publish that failed can be re-driven with POST /v1/posts/:id/retry; a dead-lettered inbound webhook with POST /v1/admin/webhook-inbox/:id/requeue.