Jobs & Queues Overview
FoundryDB gives app services two primitives for background work: Jobs and Queues. Both run inside your service's infrastructure, colocated with your data, so you do not pay for separate workers or accept extra network hops.
Jobs
A job is a container invocation that runs on your app service's VM. You define a name, the command to run, and optionally a cron schedule. The platform dispatches each fire as a transient systemd unit: resource-isolated, observable, and logged independently of your main application container.
The diagram above shows the complete lifecycle of one job invocation. A cron tick (or a manual /run call) reaches the scheduler, which dispatches an invocation that runs as a systemd-run container on the app VM. When the container exits with a non-zero code the scheduler waits retry_backoff_seconds, then re-dispatches the next attempt. Once max_retries attempts are exhausted the invocation moves to the failed terminal. An exit code of 0 at any attempt reaches succeeded immediately.
Every invocation row is persisted: status, attempt number, exit code, and the last 40 lines of log output. The previous invocation's outcome is visible before the next one fires.
Jobs at a glance
Jobs are well suited for:
- Scheduled batch processing (nightly reports, daily aggregates, periodic cleanup)
- One-shot tasks triggered from CI or an API call (database migrations, data backups, ad-hoc scripts)
- Any work that must run close to the database but does not need to stay resident
| Feature | Detail |
|---|---|
| Trigger types | Cron schedule (standard cron expression), on-demand via POST /jobs/{id}/run |
| Concurrency | Per-job cap of 1 to 5 simultaneous invocations |
| Runtime limit | Configurable from 10 seconds to 6 hours (timeout_seconds) |
| Retries | 0 to 5 attempts; retry_backoff_seconds between attempts |
| Overlap policy | skip: a cron fire that finds the cap full is recorded and skipped, never dropped silently |
| Invocation log | Last 40 lines stored on the invocation row; full log retrievable on demand |
| Limit | 20 jobs per app service |
Cron vs on-demand
A job created with a schedule field fires automatically on the cron cadence. The same job can also be fired manually at any time by calling POST /jobs/{id}/run, which creates an on-demand invocation counted against the same concurrency cap. Both paths share the same retry ceiling and log storage.
A job created without a schedule is purely on-demand and useful for deployment hooks or CI pipelines that call the FoundryDB API to run a migration or a one-off script on the VM closest to the database.
Invocation states
| State | Meaning |
|---|---|
queued | Dispatched by the scheduler; the systemd-run unit has not yet started |
running | Container is executing on the app VM |
succeeded | Container exited with code 0 |
failed | Container exited non-zero and the retry ceiling is exhausted, or the runtime limit was exceeded |
skipped | A cron fire was suppressed because the concurrency cap was already full |
Queues
A queue is a durable, PostgreSQL-backed message queue provisioned inside one of your PostgreSQL managed services. Messages live in the mdb_queue schema, transactional with your application data. Consumers claim messages with SELECT ... FOR UPDATE SKIP LOCKED so concurrent workers never race or double-process.
How competing consumers work
The key property of SKIP LOCKED is that each worker atomically claims a disjoint set of rows in a single statement. Any row already locked by another worker is skipped over, not blocked on. This means:
- Multiple workers drain the same queue table in parallel with no coordination layer outside PostgreSQL.
- A claimed row is invisible to other workers for the duration of the visibility timeout. If the claiming worker crashes without acknowledging, the row becomes visible again once the timeout expires and another worker picks it up.
- Acknowledging a message deletes the row. There is no separate state machine to drive.
- A message that exceeds
max_attemptsis moved tomdb_queue.dead_messages(the dead-letter queue, DLQ) rather than being deleted, so you retain a full audit trail of failed deliveries.
Queues at a glance
Queues are well suited for:
- Decoupling producers from consumers within your application
- Reliable task dispatch where enqueue must be atomic with a business write (enqueue in the same PostgreSQL transaction as the insert that creates the work)
- Work that needs at-least-once delivery and a dead-letter audit trail
| Feature | Detail |
|---|---|
| Backend | PostgreSQL mdb_queue schema, SKIP LOCKED row-level locking |
| Delivery | At-least-once |
| Batch enqueue | Up to 100 messages per request |
| Message size | Up to 256 KB per message |
| Visibility timeout | Configurable per queue, up to 12 hours; controls how long a claimed-but-unacknowledged message stays hidden |
| Retries | Configurable max_attempts; dead-letter queue on exhaustion |
| Limit | 50 queues per PostgreSQL service |
Delivery guarantee detail
Under normal operation each message is delivered at least once. A worker that claims a row and completes work without acknowledging (due to a crash or network partition) causes the row to reappear after the visibility timeout expires. This means your consumer logic should be idempotent where possible, or use the message's unique id column to detect and skip re-deliveries.
Messages in the DLQ (mdb_queue.dead_messages) are never automatically retried. They are available for inspection and manual replay via the API.
When to use each
| Situation | Use |
|---|---|
| Run a script every night at 02:00 | Job with cron schedule |
| Trigger a migration from your deployment pipeline | Job, invoke via POST /jobs/{id}/run |
| Fan out work from a web request to background workers | Queue |
| Audit every unit of work that succeeded, failed, or was retried | Queue with DLQ |
| Keep background work transactional with your business data | Queue (enqueue in the same transaction as the write) |
| Run a maintenance task close to the database on a fixed schedule | Job with cron schedule |
| Process webhook deliveries reliably with retries and a dead-letter trail | Queue |
How they relate
Jobs and queues are independent features. A common pattern is to combine them: a job fires on a schedule, reads from a queue, and processes a batch. The queue carries the durable work; the job drives the processing cadence. This keeps the queue from growing unbounded while still letting producers enqueue work at any rate.
Both features are scoped to a service in your FoundryDB organization. Jobs are scoped to an app service; queues are scoped to a PostgreSQL managed service.