Pipelines

Automate multi-step processes as declarative JSON DAGs: no worker code required. A pipeline definition says what to run (plugin functions, HTTP calls, timers, approvals, child workflows) and in what order; the engine executes it durably, retries failures, suspends on approvals, and renders every run on the standard workflow timeline.

Understanding Pipelines

Pipelines are built on top of the Workflows engine. Every pipeline run is an ordinary workflow run named pipeline.<name>, driven by the same event log, job queues, waits and signals machinery, so the run monitor, Gantt timeline, stats, circuit breaker, execution timeouts and crash recovery all apply to pipelines unchanged.

Use workflows when the logic lives in your own TypeScript or Python worker code. Use pipelines when the automation can be expressed as data: calling plugin functions, HTTP endpoints, LLMs, waiting on timers or human approvals, fanning out over arrays, and spawning other workflows or pipelines. Pipelines are edited in a visual builder (or raw JSON) and versioned server-side; there is nothing to deploy.

Definitions live in the pipelines system collection (inside workflows.db) and are version-bumped on every definition change. Each run pins a snapshot of the definition it started with, so editing a pipeline never mutates an in-flight run.

Creating Pipelines

From the UI

Dashboard → Pipelines → New pipeline. The builder opens with a starter definition: add steps from the canvas toolbar, connect nodes to declare dependsOn, and configure each step in the inspector that opens on click. A Canvas / JSON toggle edits the raw definition directly; both sides round-trip. Validate checks the definition server-side (per-step issues show on the badge and inside the inspector), Save stores a new version, and Run starts a manual run.

From the SDK

pipeline = await client.pipelines.create({
    "name": "note-intake",
    "description": "Index and notify on new notes",
    "steps": [
        {"id": "embed", "type": "plugin", "target": "indexer/reindex",
         "params": {"collection": "{{trigger.collection}}", "id": "{{trigger.record.id}}"}},
        {"id": "notify", "type": "http", "method": "POST",
         "url": "https://hooks.example.com/notes", "dependsOn": ["embed"]},
    ],
})

run = await client.pipelines.run("note-intake", {"priority": True})

Pipeline names match ^[a-zA-Z0-9_-]{1,64}$ and are unique per database. Patching the definition bumps version; toggling enabled does not.

The Definition

{
  "name": "note-intake",
  "description": "Index and notify on new notes",
  "enabled": true,
  "triggers": [
    { "type": "collection", "database": "default", "collection": "notes", "actions": ["create", "update"] },
    { "type": "cron", "schedule": "0 9 * * 1-5" }
  ],
  "steps": [
    {
      "id": "embed",
      "type": "plugin",
      "target": "indexer/reindex",
      "params": { "collection": "{{trigger.collection}}", "id": "{{trigger.record.id}}" },
      "timeout": 60,
      "retries": 2,
      "onError": "fail",
      "dependsOn": [],
      "condition": "{{trigger.action}} == 'create'"
    },
    { "id": "slowpath", "type": "delay", "duration": "30s", "dependsOn": ["embed"] },
    { "id": "approve", "type": "signal", "name": "approval", "dependsOn": ["embed"] },
    { "id": "vars", "type": "set", "values": { "n": "{{steps.embed.output.count}}" }, "dependsOn": ["embed"] },
    {
      "id": "notify",
      "type": "http",
      "method": "POST",
      "url": "https://hooks.example.com/notes",
      "headers": { "X-Api-Key": "..." },
      "body": { "count": "{{vars.n}}" },
      "dependsOn": ["approve", "vars"]
    },
    {
      "id": "summarize", "type": "llm",
      "model": "gpt-4o-mini",
      "system": "You write one-line summaries.",
      "prompt": "Summarize: {{trigger.record.text}}",
      "dependsOn": ["embed"]
    },
    { "id": "child", "type": "workflow", "name": "heavy-job", "input": { "ref": "{{trigger.record.id}}" }, "wait": true },
    {
      "id": "fanout", "type": "foreach",
      "items": "{{trigger.record.tags}}",
      "run": { "type": "pipeline", "name": "tag-worker", "input": { "tag": "{{item}}" } },
      "dependsOn": ["embed"]
    },
    { "id": "sub", "type": "pipeline", "name": "audit-trail", "input": {} }
  ],
  "output": { "count": "{{steps.embed.output.count}}" },
  "options": { "concurrency": 4, "maxEvents": 10000, "executionTimeout": 3600000 }
}

Triggers

Field	Values
`type: "collection"`	`database` (default `"default"`), `collection`, `actions`: subset of `create`, `update`, `delete`
`type: "cron"`	`schedule`: standard 5-field cron expression

Collection triggers fire whenever a matching record is written; no changes to the collection’s own trigger options are needed; the pipeline subscribes from its definition. Disabled pipelines ignore events.
Cron triggers are registered as in-process named schedules when the dispatcher boots (and on every definition change); they do not create rows in the cron-jobs table. A disabled pipeline’s schedules are retired.
Manual runs (UI, POST /api/pipelines/:name/runs, SDK run) always work regardless of triggers.
Token triggers are external: mint a token (POST /api/pipelines/:name/tokens) and any system can start runs with an unauthenticated POST /api/pipelines/trigger/:token; see Public trigger tokens.

The trigger context is available to every step as {{trigger.*}}:

// collection event
{ "type": "collection", "action": "create", "database": "default",
  "collection": "notes", "id": "rec_123", "record": { ... }, "timestamp": "..." }
// cron fire and manual runs
{ "type": "cron" }
// token trigger (POST /api/pipelines/trigger/:token)
{ "type": "token", "token": "ci-hook" }

Cron jobs can also start a pipeline ad-hoc with the pipeline:// command type; see Cron.

Steps

Every step shares these fields:

Field	Default	Description
`id`	-	Unique within the pipeline (`^[a-zA-Z0-9_-]{1,64}$`)
`type`	-	One of the types below
`dependsOn`	`[]`	Step ids that must complete first
`condition`	-	Expression evaluated when the step becomes ready; `false` skips the step
`retries`	`0`	Retry attempts on failure (linear backoff)
`timeout`	-	Per-step timeout in seconds
`onError`	`"fail"`	`"fail"` cancels dependents; `"continue"` records the error and moves on

`plugin`

Calls a plugin function over the warm in-process pipe (same path as the plugin:// trigger scheme, no network hop.

{ "id": "embed", "type": "plugin", "target": "indexer/reindex",
  "params": { "collection": "notes", "id": "{{trigger.record.id}}" },
  "timeout": 60, "retries": 2 }

target is <instance>/<function>; timeout is clamped to 1–120s (default 30). The function’s return value becomes the step output.

`http`

A single synchronous HTTP attempt; every attempt is logged to the webhook logs (_webhook_logs). Retries use the step-level retries (the same step_retrying mechanism as every other step type).

{ "id": "notify", "type": "http", "method": "POST",
  "url": "https://hooks.example.com/notes",
  "headers": { "X-Api-Key": "..." },
  "body": { "count": "{{vars.n}}" } }

The step output is { "status": 200, "body": ... }; body is the parsed JSON response, or the raw string for non-JSON bodies. Non-2xx statuses fail the step (with a snippet of the response in the error).

`set`

Evaluates a map of templates and folds the results into the run context as {{vars.<key>}}: pure computation, no side effects.

{ "id": "vars", "type": "set",
  "values": { "n": "{{steps.embed.output.count}}", "label": "note" } }

`delay`

Durable timer. Emits a wait_created event with the resume time; the scheduler wakes the run exactly when due, even across server restarts (the wait row survives, the resume time does not move).

{ "id": "slowpath", "type": "delay", "duration": "30s" }

duration accepts 30s, 5m, 1h, or a bare number of seconds ("90").

`signal`

Durable approval / external gate. The step parks in signal_waiting until a matching signal arrives, then completes with the signal data as its output. Signals are matched by name, first-in-first-out across the whole run event log; a signal sent before the step starts waiting also satisfies it (mirroring worker-queue semantics).

{ "id": "approve", "type": "signal", "name": "approval" }

Resume paths (all of them requeue the run):

# by run id
await client.workflow.send_signal(run_id, "approval", {"ok": True, "by": "anton"})
# or via a workflow hook token (POST /api/workflows/hooks/:token)

Downstream steps template the payload: {{steps.approve.output.ok}}.

`workflow` / `pipeline`

Spawns a child run. By default fire-and-forget: the step completes immediately with { "runId": "..." } and the pipeline continues. With wait: true the step parks (children_waiting) until the child run settles (the engine pushes a child_completed event when the child completes, so no polling, and the wait survives restarts. The child carries parentRunId and shows under the parent in the run monitor.

{ "id": "child", "type": "workflow", "name": "heavy-job",
  "input": { "ref": "{{trigger.record.id}}" },
  "wait": true, "retries": 2 }
{ "id": "sub", "type": "pipeline", "name": "audit-trail", "input": {} }

workflow children are ordinary worker workflows (a job lands on the __wkf_workflow_<name> queue for your workers); pipeline children get their own definition snapshot and run through the pipeline dispatcher.

With wait: true the step output is { "runId", "status", "output" } of the child; downstream steps template {{steps.child.output.output.*}}. A failed (or cancelled) child fails the step (respecting onError), with the child run id in the error. The child’s own executionTimeout bounds the wait; step retries apply to the spawn itself, not to re-running the child.

`foreach`

Fan-out over an array: spawns one child run per element, optionally joining on all of them.

{
  "id": "fanout",
  "type": "foreach",
  "items": "{{trigger.record.tags}}",
  "run": {
    "type": "pipeline",              // or "workflow"
    "name": "tag-worker",
    "input": { "tag": "{{item}}", "i": "{{itemIndex}}" }
  },
  "wait": true                        // default
}

items is a template (or literal array) that must resolve to an array at run time; an empty array completes immediately with empty results, a non-array fails the step. Hard cap: 1000 items per step.
Each child’s input is templated against the normal context plus {{item}} (the element, type preserved) and {{itemIndex}}.
With wait: true (the default) the step parks until every child settles. The output is { "runIds": [...], "results": [...] } with one entry per item in order, results[i] = { index, runId, status, output }, so downstream steps can read {{steps.fanout.output.results[0].output.*}}. Any failed child fails the step (respecting onError).
With wait: false the step completes immediately with { "runIds": [...], "count": n }.
Children are spawned at-least-once like every step: crash recovery and retries replay the event log and only spawn items that have no child yet, a crashed fan-out never duplicates children.

`llm`

Runs one synchronous chat completion against any OpenAI-compatible endpoint (OpenAI, Ollama, OpenRouter, vLLM, LM Studio, …).

{
  "id": "summarize",
  "type": "llm",
  "model": "gpt-4o-mini",            // default: LLM_MODEL env
  "baseUrl": "http://llm.internal:11434/v1",  // default: LLM_BASE_URL env → OpenAI
  "system": "You write one-line summaries.",
  "prompt": "Summarize: {{trigger.record.text}}",
  "temperature": 0.2,
  "maxTokens": 512,
  "jsonMode": true,
  "timeout": 120
}

Either prompt (a single user message) or messages ([{ "role", "content" }], templated), not both. Every field except messages/prompt is optional. The step output is:

{ "content": "<raw text>", "parsed": { ... },  // parsed present with jsonMode
  "model": "<served model>", "usage": { "prompt_tokens": 12, ... } }

jsonMode sends response_format: { type: "json_object" } and parses the content into output.parsed; a non-JSON reply fails the step, so downstream {{steps.summarize.output.parsed.*}} paths are safe. The API key is never stored in the definition: set "apiKeySecret": "openai_main" to name a stored secret, or fall back to the server’s LLM_API_KEY environment variable (set LLM_BASE_URL/LLM_MODEL for defaults). Failures (rate limits, timeouts) retry with the normal step retries.

Templates and conditions

Any string value in params, headers, body, input, values, condition and output supports {{dot.path}} substitution over:

Path	Contents
`trigger.*`	The trigger context (see above)
`input.*`	The run input (manual runs, spawn `input`)
`steps.<id>.output`	A completed step’s output
`steps.<id>.status`	`completed` / `failed` / `skipped`
`steps.<id>.error`	Error text of a failed-but-continued step
`vars.*`	Values folded by `set` steps
`secrets.<name>`	Decrypted stored secret values
`run.id`	The current run id

Array indexing works: {{trigger.record.tags[0]}}. A template that is the entire string preserves the value’s type (numbers, objects, arrays); templates embedded in longer strings stringify. A missing path is an error: guard with condition when a path may be absent.

Conditions are data-only expressions: comparisons == != > >= < <=, and or not, parentheses, and literals ('x', "x", numbers, true/false/null). Missing paths evaluate to null; there are no function calls, so the expression language is sandboxed by construction.

"condition": "{{trigger.action}} == 'create' and {{input.priority}} >= 2"

A false condition skips the step (step_skipped with reason condition); its dependents are skipped transitively, exactly like a failure.

Output

When the run settles, output (a map of templates) is evaluated and stored on the run. Without an output, the run output defaults to a map of the terminal steps’ outputs (steps nothing depends on).

Options

Option	Default	Description
`concurrency`	`4`	Max steps of one run executing in parallel
`maxEvents`	`1000000`	Circuit breaker: run fails if events exceed this
`executionTimeout`	engine default (365 days)	Wall-clock limit for the whole run (ms)

Execution Semantics

Readiness: a step runs when every dependency has completed. Independent branches execute in parallel (bounded by concurrency); joins simply list several dependsOn.
Serialization: runs of one pipeline execute one at a time; independent pipelines run concurrently.
Version pinning: the run snapshots the definition at trigger time into its executionContext. Later edits (or deletion) do not affect in-flight runs.
Durability: the event log is the source of truth. Steps execute at-least-once across crashes: the engine restarts, replays events, re-executes only steps that had not completed. Plugin functions and HTTP handlers should therefore be idempotent, same as with worker workflows.
Suspension: while any step waits on a delay, signal or its spawned children (wait/foreach), the run parks (no busy job). A failed branch does not fail the run while another branch is still waiting; the run settles only when every step is terminal.
Failure: with onError: "fail" (default), the failed step’s transitive dependents are skipped (step_skipped events) while unrelated branches continue; the run ends failed once no work remains. With onError: "continue", the error is captured as steps.<id>.error and dependents still run.
Replicas: replicas never dispatch pipelines or register triggers; only the master does.

Run events

Pipelines emit the standard worker event contract with correlationId = <stepId>, so the Gantt timeline renders them natively:

Event	Payload
`step_started`	`{ attempt, name, type }`
`step_completed`	`{ output }`
`step_retrying`	`{ error, attempt }`
`step_failed`	`{ error, attempt, continued? }`
`step_skipped`	`{ reason }`
`wait_created` / `wait_completed`	`{ duration, resumeAt }`
`signal_waiting` / `signal_received`	`{ name }`
`child_spawned`	`{ step, runId, index, kind, name }`
`children_waiting`	`{ count }`
`child_completed`	`{ step, runId, status, output, error? }`

Public Trigger Tokens

External systems (CI hooks, payment providers, form backends, a partner’s curl) can start runs without any vski credentials:

# one-time mint (requires the workflows role)
curl -X POST https://vski.example/api/pipelines/note-intake/tokens \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name": "github-ci"}'
# → { "id": "ptoken_...", "name": "github-ci", "created": "...", "token": "ptk_..." }

# the public trigger; no auth, the token is the credential
curl -X POST https://vski.example/api/pipelines/trigger/ptk_... \
  -H "Content-Type: application/json" -d '{"noteId": "rec_123"}'
# → 202 { "runId": "wkf_...", "workflowName": "pipeline.note-intake" }

The request body becomes the run input ({{input.*}}); the trigger context is { "type": "token", "token": "<name>" }, never the raw secret, so it is safe in event logs and run pages.
Raw tokens are returned once at mint time; only a SHA-256 hash is stored. Unknown and revoked tokens both answer 404 (no existence leak); a disabled pipeline answers 409.
The endpoint is rate limited per IP (60 requests/minute per node).
Revoke with DELETE /api/pipelines/:name/tokens/:id; the builder’s settings panel mints, displays-once and revokes tokens.
Treat tokens like passwords: rotate per integration, name them after the caller, revoke when the integration goes away.

Secrets

Pipelines need credentials (API keys for http calls, bearer tokens for LLM providers) that must not live in definitions (definitions are snapshotted into run rows and shown in the UI). The secrets store solves this: values are written through the API, encrypted at rest with AES-256-GCM, and decrypted only in memory while a run executes.

# store (admin only); storing an existing name replaces it
curl -X PUT https://vski.example/api/secrets/openai_main \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"value": "sk-..."}'
# → { "name": "openai_main", "created": "...", "updated": "..." }

# list; names and timestamps only, values are never returned
# (requires the workflows role, same as reading pipelines)
curl https://vski.example/api/secrets -H "Authorization: Bearer $TOKEN"

Runs read them in two ways:

Templates: any templated field may reference {{secrets.NAME}}; most commonly http headers:

{ "id": "sync", "type": "http", "method": "GET",
  "url": "https://api.github.com/user",
  "headers": { "Authorization": "Bearer {{secrets.github_pat}}" } }

llm steps: "apiKeySecret": "openai_main" sends the stored value as the bearer token (overriding LLM_API_KEY). A missing name fails the step loudly rather than silently sending no key.

Semantics and guardrails:

Storing and deleting secrets is admin-only. Workflows-role users can list the names (the builder’s apiKeySecret picker needs that) and reference secrets in definitions, but only admins write the store; the same stance as API keys and plugins.
Values are write-only: no endpoint returns them; replacing a secret means storing the same name again. Settings → Secrets in the dashboard manages them.
Names are letters/digits/_/- starting with a letter or _ (no dots, a dot would split the template path).
Secrets are loaded per run only when the definition references them; other runs never touch the store.
The encryption key is SECRETS_KEY when set, otherwise derived (with domain separation) from JWT_SECRET. Rotating either makes existing secrets undecryptable: runs using them fail with a clear error; set SECRETS_KEY explicitly if you rotate JWT_SECRET.
Avoid templating secrets into places that persist (for example the pipeline-level output map); step inputs like headers are the intended use; anything echoed back by a remote system is out of the platform’s control.

Monitoring Runs

Open Workflows → pipeline. (or click any run from the Pipelines list or the builder’s Runs modal). The run page shows status, input, output, the event log and the Gantt timeline; the Runs modal polls the last 10 runs with a green dot while anything is active. All the usual workflow tooling applies: stats, children listing (parentRunId), signals, hooks.

Deep link: /workflows?workflowName=pipeline.<name> pre-filters the runs list.

API Endpoints

Requires the workflows role (or admin), same as the rest of the workflow API.

Method	Endpoint	Description
`GET`	`/api/pipelines`	List pipelines
`POST`	`/api/pipelines`	Create (body = definition)
`GET`	`/api/pipelines/:name`	Get one (row + parsed definition)
`PATCH`	`/api/pipelines/:name`	Replace definition (bumps version) and/or `enabled`
`DELETE`	`/api/pipelines/:name`	Delete
`POST`	`/api/pipelines/validate`	Validate a definition (per-step issue map)
`POST`	`/api/pipelines/:name/runs`	Start a run (body = run input)
`POST`	`/api/pipelines/:name/tokens`	Mint a public trigger token (raw token returned once)
`GET`	`/api/pipelines/:name/tokens`	List trigger tokens (metadata only)
`DELETE`	`/api/pipelines/:name/tokens/:id`	Revoke a trigger token
`POST`	`/api/pipelines/trigger/:token`	Public: start a run by trigger token (no auth, rate limited)
`GET`	`/api/secrets`	List stored secrets (names and timestamps only)
`PUT`	`/api/secrets/:name`	Store/replace a secret (body `{value}`; encrypted at rest); admin only
`DELETE`	`/api/secrets/:name`	Delete a secret; admin only

Validation rejects unknown step types, duplicate ids, bad dependsOn references, cycles, malformed cron schedules, bad durations, template and condition syntax errors, and out-of-range options, each with the offending step id.

SDKs

Python (vski-sdk 0.8.0+): client.pipelines:

Method	Notes
`list()` / `get(name)` / `get_definition(name)`	`get_definition` returns the parsed definition dict
`create(definition, enabled=True, db=None)`	Definition replace bumps version; rename is rejected
`update(name, definition=None, enabled=None)`	Toggle or replace in place
`delete(name)`
`validate(definition)`	`{"valid": bool, "issues": ...}`
`run(name, input=None, run_id=None, idempotency_key=None)`	Returns the created run (`{"runId": ...}`)
`create_trigger_token(name, token_name=None)`	Returns `{"id", "name", "created", "token"}`; the raw token once
`list_trigger_tokens(name)` / `delete_trigger_token(name, token_id)`	Manage tokens
`trigger_by_token(token, input=None)`	Calls the public endpoint; returns `{"runId", "workflowName"}`

client.secrets: list(), set(name, value), remove(name).

The TypeScript SDK (@vski/sdk 2.5.0+) mirrors this surface in camelCase (client.pipelines.run(...), client.secrets.set(...)); see SDK Installation.

From the command line:

vski pipelines list
vski pipelines run note-intake
vski pipelines tokens create note-intake

Practical Examples

Index and notify on new records

{
  "name": "note-intake",
  "triggers": [
    { "type": "collection", "database": "default", "collection": "notes", "actions": ["create", "update"] }
  ],
  "steps": [
    { "id": "embed", "type": "plugin", "target": "indexer/reindex",
      "params": { "collection": "notes", "id": "{{trigger.record.id}}" }, "retries": 2 },
    { "id": "notify", "type": "http", "method": "POST",
      "url": "https://hooks.example.com/notes",
      "body": { "id": "{{trigger.record.id}}", "text": "{{trigger.record.text}}" },
      "onError": "continue", "dependsOn": ["embed"] }
  ],
  "output": { "indexed": "{{steps.embed.output.count}}" }
}

Notification failures (onError: "continue") never fail the indexing run.

Human approval gate

{
  "name": "weekly-report",
  "triggers": [{ "type": "cron", "schedule": "0 9 * * 1-5" }],
  "steps": [
    { "id": "build", "type": "plugin", "target": "reports/weekly", "params": {} },
    { "id": "approve", "type": "signal", "name": "publish-approval", "dependsOn": ["build"] },
    { "id": "publish", "type": "http", "method": "POST",
      "url": "https://cms.example.com/publish",
      "body": { "approvedBy": "{{steps.approve.output.by}}" },
      "dependsOn": ["approve"] }
  ]
}

The run parks at approve until someone sends the publish-approval signal (workflow hook URL, sendSignal, or any script), then publishes with the approver’s name templated into the request.

Fan-out to a worker workflow

{
  "name": "bulk-backfill",
  "steps": [
    { "id": "page1", "type": "workflow", "name": "heavy-job", "input": { "page": 1 } },
    { "id": "page2", "type": "workflow", "name": "heavy-job", "input": { "page": 2 } },
    { "id": "done", "type": "set", "values": { "queued": true }, "dependsOn": ["page1", "page2"] }
  ]
}

Both children are queued immediately (no dependsOn between them); done joins. Children appear linked via parentRunId.

Fan-out over an array (foreach) with join

{
  "name": "tag-enricher",
  "steps": [
    {
      "id": "fan", "type": "foreach",
      "items": "{{input.record.tags}}",
      "run": { "type": "pipeline", "name": "tag-worker", "input": { "tag": "{{item}}" } }
    },
    {
      "id": "report", "type": "set", "dependsOn": ["fan"],
      "values": { "firstRun": "{{steps.fan.output.results[0].runId}}" }
    }
  ]
}

One tag-worker run per tag, all joined before report runs; the results array carries each child’s output in item order. Set "wait": false on the foreach step to queue and move on.

LLM enrichment on new records

{
  "name": "note-summary",
  "triggers": [
    { "type": "collection", "collection": "notes", "actions": ["create"] }
  ],
  "steps": [
    {
      "id": "summarize", "type": "llm",
      "model": "gpt-4o-mini",
      "system": "You write one-line summaries.",
      "prompt": "Summarize: {{trigger.record.text}}",
      "jsonMode": true, "retries": 2, "onError": "continue"
    },
    {
      "id": "store", "type": "http", "method": "POST",
      "url": "https://cms.example.com/summaries",
      "body": { "id": "{{trigger.record.id}}", "summary": "{{steps.summarize.output.parsed.summary}}" },
      "dependsOn": ["summarize"]
    }
  ]
}

The model is configured server-side (LLM_API_KEY / LLM_BASE_URL / LLM_MODEL env); the definition only says what to ask.

Best Practices

Keep steps small and idempotent: retries and crash recovery re-execute a step that may have already had side effects.
Prefer onError: "continue" for notification-style side effects so a flaky webhook never fails the important branch.
Use condition on trigger-driven steps to separate create from update handling instead of maintaining two pipelines.
Set retries (with linear backoff) on any HTTP step that leaves your network.
Model long waits as delay/signal steps: they park the run durably instead of occupying a worker or a timeout budget.
Validate definitions in CI with POST /api/pipelines/validate; the builder’s Validate button calls the same endpoint.