Troubleshooting

Every response carries meta.requestId, echoed as the x-request-id header. Quote it — it is the only handle that ties a client-side symptom to a server-side log line.

My data is missing

The most common integration bug, and it never presents as an error.

Tenant resolution is an upsert on (appId, externalRef) — resolveOrProvision. A typo, a local database id, or last environment's workspace id all provision a brand-new empty workspace and return 200. There is no "unknown tenant" error to catch.

curl -sS "${AUTH[@]}" $INBOX/v1/whoami                    # resolves — but to WHICH tenant?
curl -sS "${AUTH[@]}" "$INBOX/v1/inbox/threads?limit=5"   # empty on a workspace you know has traffic?

If whoami returns a tenantId and the list is empty, you have provisioned a new tenant. Compare your ws_ value against the workspace's Accounts user-group id. Also check you are not using an Accounts app UUID anywhere — those differ per environment; slugs are the stable key.

Local setup failures

Cannot find module '@~inbox/types' — the service imports three workspace packages that must be compiled first. yarn build from the repo root; Turbo orders them before the service.

Missing required env var INBOX_KEK — .env lives at the repo root, not in apps/service/, and nothing in the app loads it. dotenv is a devDependency that is never imported. The yarn dev and yarn start:dev scripts pass --env-file ../../.env; running nest start directly does not. INBOX_KEK must be exactly 32 bytes (64 hex chars), and INBOX_REALTIME_JWT_SECRET at least 32 characters.

EADDRINUSE :6979 — an orphaned dist/main from a previous run. Kill the server, not the watcher: pkill -f 'dist/main'. Killing nest start leaves the re-parented child holding the port.

401 invalid_api_key

One code covers four causes, deliberately — a credential error must not tell an attacker which check failed:

• the key does not exist in Accounts;
• it has been revoked;
• it has expired;
• it introspects fine but resolves to no local app.

That last one is the surprising one, and it is usually a mismatched INBOX_ACCOUNTS_APP_SLUG. The service log names the real reason against the request id.

Also confirm the key format: the SDK throws locally on anything that does not start sk_. The legacy axs_ format is gone, and the package README still documents it.

403 insufficient_scope

On a tenant route: the credential genuinely lacks the scope. Note that admin satisfies every @RequireScopes here, so if an admin key is rejected on a tenant route you are almost certainly on an operator route without realising.

On an operator route with an admin key: admin grants nothing on /v1/operator/*. The guard checks stats and does not consult admin at all — cross-tenant reach is a separate, explicit grant so a workspace-confined admin key can never read the whole estate. Mint a key with inbox:stats; add inbox:connections:write for the mutating connection-request routes.

On an operator route for a signed-in human: a session caller must also satisfy user.isOperator, which is computed by matching INBOX_OPERATOR_APP_SLUG against the caller's Accounts apps by slug. Unset or mistyped, the expression short-circuits and every human loses operator access silently — while machine keys keep working, because they carry no user. That asymmetry is what makes it hard to spot. Slug, never id: Accounts app ids are environment-specific.

Idempotency errors

400 idempotency_key_required on a POST that only reads. The policy is driven by the HTTP verb: dataWrite: meta?.dataWrite ?? isWriteMethod(req.method). Every POST/PUT/PATCH/DELETE is a data write unless a route opts out with @ReadOnlyPost() — only /v1/campaigns/preflight, /v1/reports/query and /v1/reports/export do. Send a UUID; nothing there is unsafe to repeat.

409 idempotency_conflict. The same key arrived with a different fingerprint, computed as sha256(method + "\n" + url + "\n" + rawBody). Note that url includes the query string and rawBody is the bytes as sent — so a re-serialised body with different key ordering is a different fingerprint even when the JSON is semantically identical. Serialise once and reuse the exact string across retries.

409 idempotency_in_progress means the first request has not finished. Wait and retry; a handler that throws releases its reservation, so a genuine failure does not strand the key for 24 hours.

Sending failures

422 messaging_window_expired. The 24-hour messaging window is real. A free-form message to someone who has not messaged you inside 24 hours is refused. Open the conversation with an approved template: POST /v1/connections/:id/send-template.

422 validation_failed on ticket assign — "Assignment requires a user session; a service key cannot validate an assignee." Inbox writes the assignment but Accounts validates it; a machine key carries no user and no session token, so there is nothing to validate against. An unvalidated assignment would write an Accounts UUID that nothing can resolve. Call the route with a session.

409 conflict on opening a ticket. One open ticket per thread, enforced by a partial unique index in Postgres rather than a read-then-write check. The service does not check first — it lets the database refuse and translates the refusal into a clean 409. The application-code version of this check raced in axis-api, where two concurrent opens each read "none open" and each inserted, leaving 93 bad rows behind. If you get this, the thread already has an open ticket; fetch it rather than creating one.

Inbound webhook 401

Each provider signs differently, and the verification runs in the handler before any persistence:

Two rules cover most failures: verify against the raw body bytes (re-serialising changes the digest), and confirm the secret is the one configured for that integration. See Signature verification.

"Published but nothing happened"

The write succeeded and the row exists; the work never ran.

1. curl -s $INBOX/v1/health — if backends.redis is false you are on in-memory queues, and jobs do not survive a restart or fan out across instances.
2. Confirm a worker started. Enqueueing works from construction, but consumption needs start().
3. Grep the logs for Failed to enqueue. outbound-retry, publish-dispatch and campaign-send are best-effort on enqueue: the row is persisted and recoverable, so a failed enqueue is logged, not thrown.
4. Read the domain rows, not the queue — campaign recipients, PostDispatch rows and ImportJob.status each record their own progress. Re-drive with POST /v1/posts/:id/retry or POST /v1/admin/webhook-inbox/:id/requeue.

See Queues and workers.

Realtime connects, then nothing arrives

The socket opens, the token mints, the client reports connected — and no events ever come. That is almost always a reverse proxy that is not forwarding the WebSocket upgrade. Everything else works, because everything else is ordinary HTTP.

location /v1/realtime {
    proxy_pass http://127.0.0.1:6999;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 3600s;
}

If the proxy cannot be changed, fall back to SSE: POST /v1/realtime/sse/init then GET /v1/realtime/sse/stream.

Data written but not persisted

INBOX_DATABASE_URL did not reach the process, and the service fell back to in-memory stores without complaint. Writes succeed, reads are consistent within the process, and everything vanishes on restart.

curl -s $INBOX/v1/health
# {"status":"ok","env":"…","backends":{"database":false,…}}   ← broken

backends.database: false is the single most important line of the health check, and it is easy to miss because status still reads ok — liveness and correctness are different questions. Reports are a secondary tell: the whole reports module is Prisma-only, so those routes 404 rather than returning empty.