Scheduling Work and Running Workers

Checked against FrontMCP 1.12.2 on

Intermediate

Every job and workflow in this chapter ran because a client called execute_job or execute_workflow. Some work has no caller. The help desk's nightly report should run at night with nobody asking, expired sessions should be swept every ten minutes, and the support mailbox should be watched for as long as the server is up. FrontMCP has three things for that: a duty runs on a schedule, a workflow can have a schedule of its own, and a worker keeps a long-running process alive. All three answer a question a timer can't: when several copies of the server run, which one does the work?

You will learn

  • Why a timer in a provider runs once per copy of the server, and what a lease does about it
  • How to run work every interval or on a cron schedule with @Duty
  • How to put a workflow on a schedule, and who its runs run as
  • How to keep a long-running process supervised with @Worker, and how it reports what it sees
  • What happens to all three across several instances, at a restart, and where they don't run

A timer runs in every copy

The obvious way to sweep expired sessions every ten minutes is a timer in a provider. It works with one server. Start a second copy, as a stdio server does for each client and a deployment does for each instance, and the sweep runs twice:

Open
import { test, expect } from "@frontmcp/testing";
import { App, FrontMcpInstance } from "@frontmcp/sdk";
import { SessionSweeper, sweeps } from "./sweeper";

@App({ id: "help-desk", name: "Help Desk", providers: [SessionSweeper] })
class HelpDesk {}

test("🚩 two copies of the server, two sweeps each time", async () => {
  const config = { info: { name: "help-desk", version: "1.0.0" }, apps: [HelpDesk] };
  const first = await FrontMcpInstance.createDirect(config);
  const second = await FrontMcpInstance.createDirect(config);
  await new Promise((resolve) => setTimeout(resolve, 100));
  await first.dispose();
  await second.dispose();

  const copiesThatSwept = new Set(sweeps);
  expect(copiesThatSwept.size).toBe(2);
});

Starting FrontMCP in your browser…

FrontMCP starts when this example comes into view.

Each server builds its own SessionSweeper, and each one starts its own timer. With ten copies, the database is swept ten times every ten minutes, a nightly report is sent ten times, and two copies that compact the same file at once can corrupt it. The copies don't know about each other, so none of them can decide to stay out.

What they can share is a lease: a key in a store every copy reaches, which holds the id of one process and expires unless that process keeps renewing it. Every copy tries to take the lease, one gets it and does the work, and when that one stops, the lease runs out and another copy takes over. Duties, scheduled workflows and workers all use one.

Running work on a schedule with @Duty

A duty is a provider class with a run() method. @Duty says how often it runs and among which copies one of them runs it:

sweeper.duty.ts
import { Duty, type DutyRunContext } from "@frontmcp/sdk";
import { sessions } from "./sessions";

@Duty({ name: "sweep-sessions", every: "10m", scope: "machine" })
export class SessionSweeper {
  async run({ signal }: DutyRunContext) {
    for (const session of sessions.expired()) {
      if (signal.aborted) return; // the lease was lost, or the server is shutting down
      sessions.remove(session);
    }
  }
}
help-desk.app.ts
@App({ id: "help-desk", name: "Help Desk", providers: [SessionSweeper] })
export class HelpDeskApp {}

Register it in providers, like any provider. Every copy of the server registers the duty, and the lease chooses who runs it. Two copies started on one machine log:

Duty "sweep-sessions" runs in this process every 600000 ms
Duty "sweep-sessions" already runs elsewhere (lease "help-desk:sweep-sessions"); this one takes it over if that one stops
OptionWhat it says
everyHow often: milliseconds, or "500ms", "30s", "10m", "1h", "1d"
cron, timezoneOr when, on the calendar: see below
scopeAmong which copies one runs it: "machine", "workspace" (the copies of one machine that run from the same project folder) or "cluster" (every machine that shares the server's Redis)
nameThe lease's name. The class name by default; set it, since a build can rename classes

run() gets a signal that aborts when this process loses the lease or the server shuts down. A long run should look at it between steps, or pass it on to what it awaits. A run that throws is logged, Duty "sweep-sessions" failed: <cause>, and the duty runs again at its next time. A run that would start while the last one is still going is skipped.

A duty is a class, so a test can run it without waiting ten minutes. The Playground can't start the duty, but it can read what the decorator resolved, with dutyMetadataOf(), and call run() itself:

Open
import { test, expect } from "@frontmcp/testing";
import { dutyMetadataOf } from "@frontmcp/sdk";
import { SessionSweeper } from "./sweeper.duty";
import { sessions } from "./sessions";

const threeSessions = () => [
  { id: "s-1", expired: true },
  { id: "s-2", expired: false },
  { id: "s-3", expired: true },
];

test("the decorator resolved the schedule: every 600000 ms, in one process of the machine", () => {
  expect(dutyMetadataOf(new SessionSweeper())).toEqual({ name: "sweep-sessions", everyMs: 600000, scope: "machine", leaseMs: 60000 });
});

test("one run removes the expired sessions", async () => {
  sessions.reset(threeSessions());
  await new SessionSweeper().run({ signal: new AbortController().signal });
  expect(sessions.ids()).toEqual(["s-2"]);
});

test("a run whose signal is aborted stops before the next session", async () => {
  sessions.reset(threeSessions());
  await new SessionSweeper().run({ signal: AbortSignal.abort() });
  expect(sessions.ids()).toEqual(["s-1", "s-2", "s-3"]);
});

Starting FrontMCP in your browser…

FrontMCP starts when this example comes into view.

leaseMs is how long the lease outlives a holder that stopped renewing it. It defaults to three intervals, kept between 3 and 60 seconds, and decides how soon another copy takes over from one that crashed.

On the calendar, with cron

An interval drifts with every restart and knows nothing about weekdays. For a report at nine on weekday mornings, give the duty a cron expression in place of every, and the time zone it is meant in:

sla-report.duty.ts
import { Duty, type DutyRunContext } from "@frontmcp/sdk";

@Duty({ name: "sla-report", cron: "0 9 * * 1-5", timezone: "Europe/Berlin", scope: "machine" })
export class SlaReport {
  async run({ signal }: DutyRunContext) {
    // Find the tickets that broke their SLA and mail the report.
  }
}

The five fields are minute hour day-of-month month day-of-week; a sixth, in front, is seconds.

ExpressionRuns
0 9 * * 1-5At 09:00, Monday to Friday
*/15 * * * *Every 15 minutes
0 0 1 * *At midnight on the first of each month
*/10 * * * * *Every 10 seconds
@dailyAt midnight

A cron duty behaves differently from an interval in two ways that matter at a restart:

  • It does not run when the server starts. It waits for its next scheduled time.
  • It catches up once. The lease store remembers when the schedule was last taken. A copy that gets the lease after a scheduled time passed, because every copy was down at 09:00 or the holder crashed, runs once at once, however many times were missed, then continues on schedule.

Putting a workflow on a schedule

The nightly report from Chaining Jobs into Workflows is a workflow, and so far a client had to run it. A workflow has a trigger, and "schedule" starts a background run at each time of a cron:

nightly-report.workflow.ts
import { Workflow, z } from "@frontmcp/sdk";

@Workflow({
  name: "nightly-ticket-report",
  trigger: "schedule",
  schedule: { cron: "0 2 * * *", timezone: "Europe/Berlin", input: { priority: "high" } },
  inputSchema: { priority: z.enum(["high", "normal"]) },
  steps: [
    { id: "breaches", jobName: "find-sla-breaches" },
    { id: "report", jobName: "write-report", dependsOn: ["breaches"], input: (steps) => ({ breaches: steps.get("breaches").outputs.breaches }) },
  ],
})
export class NightlyTicketReport {}

schedule.input is the input of every scheduled run, checked against the inputSchema like a client's. The server holds one lease per schedule, so each scheduled time starts one run, however many copies are up:

The schedule of workflow "nightly-ticket-report" runs in this process on cron "0 2 * * *" (Europe/Berlin)

Should the report be a cron duty or a scheduled workflow? Use the workflow when the work is several jobs, or when you want what jobs give you: retries, a run with an id and a state, and execute_workflow to run it by hand as well. Use a duty for plain recurring work in a provider.

Who a scheduled run runs as

A run that a client starts runs as that client. A scheduled run has no client: it runs as the role schedule:<workflow name>, and nothing else. That matters as soon as the workflow, or a job it runs, has permission rules:

permissions: [
  { action: "execute", roles: ["admin", "schedule:nightly-ticket-report"] }, // admins by hand, and the schedule
  { action: "read", roles: ["admin"] }, // nobody owns a scheduled run, so say who may read it
],

Without the schedule's role among the execute roles, every scheduled time is refused and logged, and nothing runs:

The schedule did not start workflow "nightly-ticket-report": its create or execute rules don't allow the role "schedule:nightly-ticket-report"; add it to their roles

A webhook's and an event's runs work the same way, as webhook:<workflow> and event:<event>.

Keeping a process running with @Worker

A duty runs, ends, and runs again. Watching the support mailbox never ends: the connection stays open and mail arrives when it arrives. Started from a provider's onReady(), such a loop has nobody to start it again when the connection drops, and nobody to stop it at shutdown. A worker is a provider class whose run() the server supervises:

inbox.worker.ts
import { Worker, type WorkerRunContext } from "@frontmcp/sdk";
import { Mailbox } from "./mailbox";

@Worker({ name: "inbox-watcher", scope: "machine" })
export class InboxWatcher {
  constructor(private readonly mailbox: Mailbox) {}

  async run({ signal }: WorkerRunContext) {
    for await (const mail of this.mailbox.incoming(signal)) {
      this.mailbox.openTicket(mail);
    }
  }
}

The server does four things with it:

  1. Starts run() once the server serves. With a scope other than "process", in the one copy that holds the worker's lease.
  2. Runs it again when it ends. When run() throws or returns before its signal aborted, the server waits and runs it again: 1 second the first time, then twice as long each time, up to a minute. restart: { minDelayMs, maxDelayMs, factor } sets the waits, and restart: false runs it once.
  3. Keeps the failure away from the server. What run() throws is logged with the worker's name. It doesn't crash the server or stop other workers.
  4. Stops it at shutdown. The server aborts signal and waits for run() to return, up to 3 seconds by default.

A mailbox whose connection keeps dropping looks like this in the log:

Worker "inbox-watcher" runs in this process
Worker "inbox-watcher" failed: connection lost; it runs again in 1000 ms
Worker "inbox-watcher" failed: connection lost; it runs again in 2000 ms
Worker "inbox-watcher" failed: connection lost; it runs again in 4000 ms

So run() has two obligations, and both can be tested by calling it: return when the signal aborts, and leave by throwing when the work fails. Don't catch the error and loop inside run(): the server already waits, waits longer each time, and reports the state.

Open
import { test, expect } from "@frontmcp/testing";
import { workerMetadataOf } from "@frontmcp/sdk";
import { InboxWatcher } from "./inbox.worker";
import { Mailbox } from "./mailbox";

test("the defaults: one process of the machine, restarts from 1 s up to 60 s", () => {
  expect(workerMetadataOf(new InboxWatcher(new Mailbox()))).toEqual({
    name: "inbox-watcher",
    scope: "machine",
    leaseMs: 15000,
    restart: { minDelayMs: 1000, maxDelayMs: 60000, factor: 2 },
  });
});

test("run() opens a ticket for each mail until its signal aborts, then returns", async () => {
  const mailbox = new Mailbox();
  const shutdown = new AbortController();
  const running = new InboxWatcher(mailbox).run({ signal: shutdown.signal });

  mailbox.deliver("Cannot log in");
  mailbox.deliver("Invoice is wrong");
  await new Promise((resolve) => setTimeout(resolve, 10));
  expect(mailbox.tickets).toEqual(["Cannot log in", "Invoice is wrong"]);

  shutdown.abort();
  await running;
  expect(mailbox.connected).toBe(false);
});

test("a dropped connection leaves run() by throwing, which is what makes the server run it again", async () => {
  const mailbox = new Mailbox();
  const running = new InboxWatcher(mailbox).run({ signal: new AbortController().signal });

  mailbox.drop();
  await expect(running).rejects.toThrow("connection lost");
  expect(mailbox.connected).toBe(false);
});

Starting FrontMCP in your browser…

FrontMCP starts when this example comes into view.

scope has one more value for a worker: "process", where every copy runs its own, with no lease. Use it for work each copy needs for itself, like keeping a cache in its own memory warm. Use a leased scope for anything there should be one of, like the mailbox connection.

Telling clients, and workflows, what a worker saw

A worker that only writes to a database is invisible to the rest of the server. To tell someone, it publishes an event: clients subscribed to it are notified, and a workflow with trigger: "event" starts with the occurrence as its input. The worker takes the server's publisher from the EVENT_PUBLISHER token, through a factory provider:

inbox.worker.ts
import { AsyncProvider, EVENT_PUBLISHER, Worker, type EventPublisher, type WorkerRunContext } from "@frontmcp/sdk";
import { mailReceived } from "./mail-received.event";
import { Mailbox } from "./mailbox";

@Worker({ name: "inbox-watcher", scope: "machine" })
class InboxWatcher {
  constructor(
    private readonly mailbox: Mailbox,
    private readonly events: EventPublisher,
  ) {}

  async run({ signal }: WorkerRunContext) {
    for await (const mail of this.mailbox.incoming(signal)) {
      await this.events.publish(mailReceived, { from: mail.from, subject: mail.subject });
    }
  }
}

export const inboxWatcher = AsyncProvider({
  provide: InboxWatcher,
  name: "inbox-watcher",
  inject: () => [Mailbox, EVENT_PUBLISHER] as const,
  useFactory: (mailbox, events) => new InboxWatcher(mailbox, events),
});

Register inboxWatcher in providers, and the event in the app's events. A duty publishes the same way. The Playground below has no worker, so a tool stands in for it and publishes what the worker would. The workflow doesn't know the difference: it listens to the event's name.

Open
import { App, Job, JobContext, Tool, ToolContext, Workflow, event, z } from "@frontmcp/sdk";

export const tickets: { from: string; subject: string }[] = [];

export const mailReceived = event({
  name: "mail.received",
  description: "A mail arrived in the support mailbox",
  payloadSchema: { from: z.string(), subject: z.string() },
});

// What a published event looks like on the way in: the occurrence, with the payload under `data`.
const occurrence = {
  eventId: z.string(),
  name: z.string(),
  timestamp: z.string(),
  data: z.object({ from: z.string(), subject: z.string() }),
};

@Job({ name: "open-ticket", inputSchema: occurrence, outputSchema: { opened: z.boolean() } })
class OpenTicket extends JobContext {
  async execute({ data }: { data: { from: string; subject: string } }) {
    tickets.push(data);
    return { opened: true };
  }
}

@Workflow({
  name: "triage-mail", trigger: "event", event: "mail.received",
  inputSchema: occurrence,
  steps: [{ id: "open", jobName: "open-ticket" }],
})
class TriageMail {}

// Stands in for the worker: it publishes what the worker would when a mail arrives.
@Tool({ name: "receive_mail", description: "Pretend a mail arrived in the support mailbox", inputSchema: { from: z.string(), subject: z.string() } })
class ReceiveMail extends ToolContext {
  async execute(mail: { from: string; subject: string }) {
    return this.events.publish(mailReceived, mail);
  }
}

@App({ id: "help-desk", name: "Help Desk", tools: [ReceiveMail], events: [mailReceived], jobs: [OpenTicket], workflows: [TriageMail] })
export class HelpDesk {}

Starting FrontMCP in your browser…

FrontMCP starts when this example comes into view.

publish() answers with the occurrence's eventId and matched, the number of webhook subscriptions it was queued for. The workflow starts either way, and its run runs as the role event:mail.received. With a leased worker, one copy publishes; with scope: "process", every copy publishes its own occurrence of the same mail.

Several copies, restarts, and where it runs

What scope to pick depends on where the copies are:

The copies runA duty or worker there should be one ofNeeds
On one machine: a stdio server per client, several processesscope: "machine", or "workspace" for one per project folderThe @frontmcp/storage-sqlite package, for the lease file under ~/.frontmcp, unless the server has a redis
On several machines, containers or podsscope: "cluster"A top-level redis on the server, the same on every instance

A scheduled workflow has no scope: with redis its schedule is shared by every instance, and without it by the copies of one machine that run from the same project folder.

What happens when things stop and start:

A dutyA scheduled workflowA worker
The holder shuts down (SIGTERM)It aborts the run, waits for it and releases the lease; another copy takes over at its next checkThe sameThe same
The holder crashesAnother copy takes over once the lease expires: leaseMs plus a third of itThe same, within 80 secondsThe same, within 20 seconds by default
Every copy was down at a scheduled timeCron: one run when a copy is back. Interval: a run at once when the interval since the last run has passedOne run when a copy is backIt starts again
The lease store can't be openedThe server starts, logs it, and tries again; the duty runs once it opensThe sameThe same

Each started duty and worker also has a check in /readyz, duty:<name> and worker:<name>, which says whether this instance holds it and how the worker is doing:

{"duty:sweep-sessions":{"status":"healthy","details":{"holdsLease":false}},"worker:inbox-watcher":{"status":"degraded","details":{"state":"restarting","restarts":3,"lastError":"connection lost"},"error":"connection lost"}}

All of this needs a process that stays up:

  • A long-lived Node server, in a container or under a service manager, runs them for as long as it runs.
  • A serverless or edge deployment starts none of them. A function that lives for one request has nothing to keep a timer in. The server logs Duty "…" does not run in a serverless or edge deployment: it needs a long-lived process, and the same for workers and scheduled workflows.
  • A stdio server that a client starts runs them only while that client is connected.
  • The shared local server stays up while a session is attached, and exits idleTimeoutMs, ten minutes by default, after the last one left. A running worker doesn't count as activity. Pass idleTimeoutMs: false to runShared() for one that must keep running.

The references have the rest: @Duty the cron fields, the lease keys and the store, @Worker the states /readyz reports and watching files with watchFiles(), and @Workflow the schedule options. High availability has the table of what runs where.

Recap

  • A timer in a provider runs in every copy of the server. A duty, a scheduled workflow and a leased worker run in one, the copy that holds a lease, and another copy takes over when that one stops.
  • @Duty({ every | cron, timezone, scope }) on a provider class with run({ signal }) runs recurring work. A cron duty doesn't run at startup, reads its expression in UTC unless timezone says otherwise, and catches up once after a missed time.
  • @Workflow({ trigger: "schedule", schedule: { cron, timezone, input } }) starts a background run at each scheduled time, as the role schedule:<workflow>: add it to the workflow's execute rules when it has any.
  • @Worker({ scope, restart }) keeps run({ signal }) going: started once the server serves, run again with a growing wait when it throws or returns, stopped at shutdown. Return when signal aborts, and throw when the work fails.
  • A worker or a duty publishes events through EVENT_PUBLISHER, and a workflow with trigger: "event" starts for each one.
  • On one machine use scope: "machine" or "workspace"; across machines use "cluster", with redis. Nothing here runs on serverless or edge, and /readyz shows each one as duty:<name> or worker:<name>.

Try some challenges

Each challenge runs hidden checks against your code. Edit the code, then press Check.

Read all challenge instructions

Send the report on weekday mornings

This challenge is open below.

Stop when the server says so

This worker takes tickets off a queue. It never returns, so the server waits out its shutdown timeout every time it stops, and a copy that loses the worker's lease keeps consuming next to the copy that took over. Make run() return when its signal aborts, and close the queue when it ends, however it ends. queue.next(signal) resolves with undefined once that signal aborts.

import { Worker, type WorkerRunContext } from "@frontmcp/sdk";
import { TicketQueue } from "./queue";

@Worker({ name: "ticket-queue", scope: "machine" })
export class TicketQueueConsumer {
  constructor(private readonly queue: TicketQueue) {}

  async run(_context: WorkerRunContext) {
    while (true) {
      const ticket = await this.queue.next();
      if (ticket !== undefined) this.queue.handled.push(ticket);
    }
  }
}
/** A stand-in for a queue connection. */
export class TicketQueue {
  readonly handled: string[] = [];
  closed = false;
  private waiting: string[] = [];
  private wake: () => void = () => {};

  add(ticket: string) {
    this.waiting.push(ticket);
    this.wake();
  }

  /** The next ticket, or `undefined` once `signal` aborts. */
  async next(signal?: AbortSignal): Promise<string | undefined> {
    signal?.addEventListener("abort", () => this.wake(), { once: true });
    while (this.waiting.length === 0 && !signal?.aborted) {
      await new Promise<void>((resolve) => (this.wake = resolve));
    }
    return signal?.aborted ? undefined : this.waiting.shift();
  }

  close() {
    this.closed = true;
  }
}

Let the event start the workflow

Mail arrives, mail.received is published, and no ticket is opened: the triage-mail workflow only lets the support role run it, and a run that an event starts has no such role. Let the event start the workflow, without letting everyone else run it by hand. The Logs tab says why each run is refused.

import { Workflow, z } from "@frontmcp/sdk";

export const occurrence = {
  eventId: z.string(),
  name: z.string(),
  timestamp: z.string(),
  data: z.object({ from: z.string(), subject: z.string() }),
};

@Workflow({
  name: "triage-mail", trigger: "event", event: "mail.received",
  inputSchema: occurrence,
  permissions: [{ action: "execute", roles: ["support"] }],
  steps: [{ id: "open", jobName: "open-ticket" }],
})
export class TriageMail {}
import { App, Job, JobContext, Tool, ToolContext, event, z } from "@frontmcp/sdk";
import { TriageMail, occurrence } from "./triage.workflow";

export const tickets: { from: string; subject: string }[] = [];

export const mailReceived = event({
  name: "mail.received",
  description: "A mail arrived in the support mailbox",
  payloadSchema: { from: z.string(), subject: z.string() },
});

@Job({ name: "open-ticket", inputSchema: occurrence, outputSchema: { opened: z.boolean() } })
class OpenTicket extends JobContext {
  async execute({ data }: { data: { from: string; subject: string } }) {
    tickets.push(data);
    return { opened: true };
  }
}

// Stands in for the worker that watches the mailbox.
@Tool({ name: "receive_mail", description: "Pretend a mail arrived in the support mailbox", inputSchema: { from: z.string(), subject: z.string() } })
class ReceiveMail extends ToolContext {
  async execute(mail: { from: string; subject: string }) {
    return this.events.publish(mailReceived, mail);
  }
}

@App({ id: "help-desk", name: "Help Desk", tools: [ReceiveMail], events: [mailReceived], jobs: [OpenTicket], workflows: [TriageMail] })
export class HelpDesk {}

Challenge 1 of 3

Send the report on weekday mornings

The SLA report runs every 24 hours, at whatever time of day it first ran, weekends included, and once per machine. The team wants it at 09:00 from Monday to Friday, Berlin time, and once among all the instances, which share a Redis. Change the duty's options; the tool shows what the decorator resolves.

Open
import { Duty, type DutyRunContext } from "@frontmcp/sdk";

@Duty({ name: "sla-report", every: "1d", scope: "machine" })
export class SlaReport {
  async run({ signal }: DutyRunContext) {
    // Find the tickets that broke their SLA and mail the report.
  }
}

Starting FrontMCP in your browser…

FrontMCP starts when this example comes into view.

Report a docs issue · Report a security concern privately