Jobs

Defining Jobs

Jobs are background tasks defined in the jobs/ directory:

// jobs/send-welcome-email.js
export default {
  handler: async (ctx) => {
    const { mail, log } = ctx;
    const { email, name } = ctx.payload;
    await mail.send({
      to: email,
      subject: `Welcome, ${name}!`,
      html: `<h1>Welcome to our app, ${name}!</h1>`,
    });
    log.info('Welcome email sent', { email });
  },
  retries: 3,
};

Job handlers receive a context object with infrastructure services plus payload:

{
  db, events, queue, cache, files, mail, log, // infrastructure
  payload, // the data passed when the job was enqueued
}

Job Definition Options

export default {
  name: 'my-job',                    // Optional: unique job identifier (defaults to file path)
  handler: async (ctx) => {},        // Required: job logic
  schedule: '0 9 * * *',            // Optional: cron expression or 'continuous'
  retries: 3,                        // Optional: max retry attempts (default: 0)
  maxConcurrency: 1,                 // Optional: max parallel executions
  cooldownMs: 5000,                  // Optional: minimum ms between continuous iterations (default: 1000)
  staleTimeout: 10 * 60 * 1000,     // Optional: ms before a stuck job is recovered (default: 5 min)
  throughput: {                      // Optional: rate limiting
    maxPerSecond: 10,
    maxPerMinute: 100,
  },
};

Enqueuing Jobs

Jobs without a schedule can be triggered from route handlers or listeners:

export const POST = {
  handler: async (ctx) => {
    const { queue } = ctx;
    await queue.push('send-welcome-email', {
      email: ctx.req.body.email,
      name: ctx.req.body.name,
    });
    return { data: { queued: true } };
  },
};

Scheduled Jobs (Cron)

// jobs/generate-invoice.js
export default {
  name: 'generate-invoice',
  schedule: '0 9 * * *', // Every day at 9:00 AM
  handler: async (ctx) => {
    const { db } = ctx;
    const users = await db('users').where('plan', 'paid').select('*');
    for (const user of users) {
      await generateInvoice(db, user);
    }
  },
};

Cron format: minute hour day-of-month month day-of-week

Continuous Jobs

For jobs that need to run in an endless loop (e.g. queue consumers, polling workers), use schedule: 'continuous':

// jobs/process-webhooks.js
export default {
  schedule: 'continuous',
  cooldownMs: 5000, // wait at least 5s between iterations
  handler: async (ctx) => {
    const { db } = ctx;
    const items = await db('webhook_queue').where('status', 'pending').limit(10);

    for (const item of items) {
      await processWebhook(item);
      await db('webhook_queue').where('id', item.id).update({ status: 'done' });
    }
  },
};

Behavior:

• Handler is called immediately on boot and repeatedly — when one call completes, the next starts
• cooldownMs (default: 1000) sets the minimum time between iterations. If the handler finishes in 200ms with a 5000ms cooldown, the framework sleeps 4800ms before the next call. If the handler takes longer than cooldownMs, the next iteration starts immediately — no artificial delay is added
• Set cooldownMs: 0 to disable the cooldown and run back-to-back with no delay
• On error: exponential backoff (1s, 2s, 4s, 8s...) resets after a successful call
• Graceful shutdown: loops terminate when the server shuts down

Retries and Backoff

When a job fails and has retries configured, it's automatically retried with exponential backoff:

export default {
  name: 'charge-card',
  retries: 5, // Total attempts: retries + 1 = 6
  handler: async (ctx) => {
    await chargeCard(ctx.payload);
  },
};

Backoff schedule: baseMs * 2^(attempt - 1) — e.g., 1s, 2s, 4s, 8s, 16s.

Jobs stuck in "running" state are automatically recovered back to pending. The default timeout is 5 minutes, configurable per job with staleTimeout:

export default {
  handler: async (ctx) => { /* long-running work */ },
  staleTimeout: 30 * 60 * 1000, // allow up to 30 minutes before recovery
};

You can also set a global default via the jobs config staleTimeoutMs.

Concurrency and Throughput

Control how many instances of a job run simultaneously and how fast:

export default {
  name: 'send-notification',
  maxConcurrency: 3,     // At most 3 running at once
  throughput: {
    maxPerSecond: 10,     // Rate limit
    maxPerMinute: 500,
  },
  handler: async (ctx) => {
    await sendPush(ctx.payload);
  },
};

Concurrency is enforced via a lease system — each running instance holds a lease with a heartbeat. Leases expire after 2 minutes if the process dies.

Worker Threads

App-scope job handlers run in a pool of Node.js worker threads by default, so slow or CPU-heavy jobs don't block the HTTP server or other handlers running on the main thread.

// arcway.config.js
export default {
  jobs: {
    workerPoolSize: 3, // default: availableParallelism() - 1
  },
};

Behavior:

• workerPoolSize defaults to availableParallelism() - 1 so one core stays free for the main thread (HTTP, polling, dispatch). Set to 0 to disable worker threads entirely and run every handler inline.
• Each worker lazily builds its own infrastructure — db, redis, queue, cache, files, mail, events, log — from the same config used by the main process. The first job a worker runs pays the setup cost; subsequent jobs on the same worker reuse the warmed services.
• The handler file is dynamic-imported inside the worker (by path), so handlers must be resolvable from disk at runtime. This is the normal case for files under jobs/.
• staleTimeout doubles as the per-task worker timeout. If a handler exceeds it, the worker is terminated, a replacement is spawned, and the job is retried per its retries setting.
• Continuous jobs always run inline on the main thread, regardless of workerPoolSize. They're long-lived loops and not a fit for the per-task worker model.
• System jobs (framework-internal housekeeping) also run inline.
• Workers shut down gracefully when the server stops — in-flight tasks get a chance to finish before the pool closes.

Job Drivers

• memory — in-process, lost on restart. Good for development.
• knex (default) — persisted to the database. Survives restarts. Supports distributed workers.