VSKI v0.15.0

Pipelines v2: foreach fan-out, child waits, public trigger tokens, llm steps, encrypted secrets

v0.14.0 shipped pipelines with a deliberate “not yet” list. v0.15.0 clears five of them: fan-out over arrays, waiting on spawned children, external token triggers, a native LLM step, and an encrypted secrets store for pipeline credentials. (The realtime canvas overlay was since solved differently and stays out.)

foreach — fan-out over arrays

{
  "id": "fan", "type": "foreach",
  "items": "{{trigger.record.tags}}",
  "run": { "type": "pipeline", "name": "tag-worker", "input": { "tag": "{{item}}" } }
}

One child run per element of the resolved array (template or literal, cap 1000). Each child’s input is templated against the normal context plus {{item}} (the element, type preserved) and {{itemIndex}}. By default the step joins: it parks until every child settles and its output is { runIds, results } with one { index, runId, status, output } entry per item — downstream steps read {{steps.fan.output.results[0].output.*}}. "wait": false queues and moves on. Any failed child fails the step (respecting onError). Fan-out is at-least-once like every step: crash recovery replays the event log and spawns only items without a child yet — a crashed fan-out never duplicates children.

wait: true — join on spawned children

workflow and pipeline steps stay fire-and-forget by default; set wait: true and the step parks (children_waiting) until the spawned run settles. Completion is pushed, not polled: when a child run reaches a terminal status — pipeline settle, worker PATCH, cancellation — the engine writes a child_completed event onto the parent’s log and requeues it, so the wake-up survives restarts and works identically for worker workflows and child pipelines. The step output is { runId, status, output } of the child; a failed or cancelled child fails the step with the child run id in the error.

Public trigger tokens

External systems can now start runs without any vski credentials:

curl -X POST https://vski.example/api/pipelines/note-intake/tokens \
  -H "Authorization: Bearer $KEY" -d '{"name": "github-ci"}'
# → { "id": "ptoken_...", "token": "ptk_..." }   (raw token shown once)

curl -X POST https://vski.example/api/pipelines/trigger/ptk_... -d '{...}'
# → 202 { "runId": "wkf_..." }                   (no auth; token is the credential)

The request body becomes the run input; the trigger context is { "type": "token", "token": "github-ci" } — the raw secret never lands in the event log or run pages. Only a SHA-256 hash is stored; unknown and revoked tokens both answer 404, disabled pipelines 409; the endpoint is rate limited per IP (60/min per node). The builder’s settings panel mints, displays-once and revokes tokens.

llm — native LLM step

{
  "id": "summarize", "type": "llm",
  "model": "gpt-4o-mini",
  "system": "You write one-line summaries.",
  "prompt": "Summarize: {{trigger.record.text}}",
  "jsonMode": true, "retries": 2
}

One synchronous chat completion against any OpenAI-compatible endpoint — OpenAI, Ollama, OpenRouter, vLLM, LM Studio. baseUrl/model fall back to the LLM_BASE_URL/LLM_MODEL env vars (then OpenAI defaults); the API key resolves from the step’s apiKeySecret (a stored secret, see below) or the server’s LLM_API_KEY — it never lives in a definition. Output: { content, parsed?, model, usage } — with jsonMode the reply is parsed into output.parsed (a non-JSON reply fails the step, so downstream paths are safe). messages replaces prompt for full threads; retries and timeouts reuse the standard step machinery.

Encrypted secrets store

Credentials for http headers and LLM providers no longer need to sit in definitions (which are snapshotted into run rows and rendered in the UI):

curl -X PUT https://vski.example/api/secrets/openai_main \
  -H "Authorization: Bearer $KEY" -d '{"value": "sk-..."}'

Values are AES-256-GCM encrypted at rest (key: SECRETS_KEY, or derived with domain separation from JWT_SECRET) and write-only — no endpoint returns them. Storing and deleting secrets is admin-only (workflows-role users can list names and reference secrets in definitions, same stance as API keys and plugins). Runs decrypt them in memory only when the definition references them: {{secrets.NAME}} in any templated field, or llm apiKeySecret. A rotated key fails affected runs with a clear message rather than returning junk. The dashboard gains Settings → Secrets, and the llm inspector offers an apiKeySecret picker fed by the store.

Also in this release

• New run events on the pipeline timeline: child_spawned ({ step, runId, index, kind, name }), children_waiting, and child_completed ({ step, runId, status, output }).
• The builder canvas palette and inspector learn the two new step types, the wait toggle, a trigger-token manager, and an llm apiKeySecret picker; a new Settings → Secrets page manages the store.
• Validation covers the new surface: foreach run specs (type, name, self-reference), llm field ranges and apiKeySecret names, and their template syntax.

Tests

test/e2e/55_pipelines_test.ts grows to 14 steps: foreach fan-out with join over three pipeline children (per-item templating, ordered results, parentRunId links), wait: true on a workflow child unblocked by a worker-style run PATCH, the full token lifecycle against the unauthenticated endpoint (404 for unknown/revoked, 409 disabled, input passthrough, non-leaking trigger context), an llm step verified end-to-end against an in-test OpenAI-compatible mock (request shape, system+user messages, response_format, usage passthrough, parsed templating), and the secrets store ({{secrets.*}} in http headers and set values, llm apiKeySecret sent as the bearer, write-only listings, delete, and the admin-only guard: a workflows-role API key gets 403 on store/delete but can still list). Go unit tests extend the executor suite (fan-out/park/join, no-respawn across requeues, failed children, wait semantics, llm request shaping, secrets in the template context, secrets not loaded for plain definitions) and add token, child-completion-hook and secrets service tests over SQLite (encryption at rest, key-rotation failure, llm key resolution order) plus a handler test for the admin-only mutations.

Compatibility

• Additive: two new step types, optional wait field, new pipeline_tokens and secrets tables (workflows.db, registered as system collections), new endpoints and SDK methods. Existing definitions and runs are unaffected; in-flight runs keep their pinned snapshots.
• foreach joins by default ("wait": false opts out); spawn steps remain fire-and-forget by default ("wait": true opts in).
• New env vars (all optional): LLM_API_KEY, LLM_BASE_URL, LLM_MODEL, SECRETS_KEY. llm steps fail with a clear message when no key is available against the default OpenAI endpoint. Secrets encrypt under SECRETS_KEY or (derived) JWT_SECRET — rotating either invalidates stored secrets, so set SECRETS_KEY explicitly if JWT_SECRET rotates.
• JS SDK 2.5.0 and Python SDK 0.8.0 publishing is part of the release run (client.pipelines.createTriggerToken / triggerByToken and client.secrets.set/list/remove, Python equivalents); the web builder calls the token and secrets endpoints directly so it works with any published SDK version.

What’s next

Still open from the original list: compensation/rollback steps. Ideas under consideration: per-foreach parallelism limits, cancellation propagation from a failed parent to its children.