Skip to content

VSKI v0.17.5

The plugin platform grows up in two directions: plugins can now define their own pipeline step types (a container step instead of the start → signal → check workaround) and report live stats; both process metrics the server collects itself and custom metrics the plugin publishes. Under the hood, the storage layer fixes a connection-pool race that could make pooled connections read through deleted WAL files (empty/stale reads, then database disk image is malformed on writes); that class of failure is now structurally impossible from inside the server and blocked from outside it.

Safe upgrade: no schema migrations; stats tables are created in stats.db on start and plugin catalogs are re-scanned automatically. Everything is on by default. SDKs: @vski/sdk 2.10.0, vski-sdk 0.15.0.

A plugin manifest can now declare step types alongside its functions:

steps:
container:
description: Build/run an ephemeral Docker container and report its exit code and output.
params: containers/step.schema.json # file path or inline JSON Schema
functions:
start: container.start # required; must ack fast
check: container.check # optional, informational
  • Registry and validation. Scanning inlines the params schema into the catalog; step types resolve per database from plugins with an enabled instance. Type names must match ^[a-zA-Z0-9_-]{1,64}$ and collisions across plugins are rejected at validation. Param templates ({{...}}) are checked as usual; fully-resolved params are validated against the plugin’s JSON Schema (draft 2020-12).
  • Asynchronous execution. The server dispatches the start function with the step params plus an injected context (runId, stepId, pipeline, db, signal, timeout, attempt) under a 30-second dispatch budget: the function acks and does its work in the background. The step parks on a deterministic per-step signal; the plugin completes it with client.workflow.send_signal(runId, signal, {ok, error, output}): ok: false fails the step with error, anything else completes it with output. Completion is durable: signals that arrive before the park are buffered and replayed, runId/stepId give the plugin an idempotency key for re-dispatches, and the step timeout (1–86400 s) is passed through as the overall budget. Retries cover dispatch failures; a plugin-reported failure is terminal.
  • Builder. Plugin step types appear in the palette next to the builtins with their own icon and accent color, and the inspector renders a form from the params schema (typed inputs, enums as selects, JSON fallback for complex shapes).
  • The containers plugin ships a native container step: build/run an ephemeral container with resource caps, log capture and timeout enforcement, replacing the three-step workaround in a single node.

Every plugin instance gets a live stats view, from two sources merged by the 30-second stats collector:

  • Process stats (server-collected). Read from /proc per plugin child process: RSS, CPU % (utime+stime delta), open file descriptors, uptime, stored in _stats_plugin_processes.
  • Custom metrics (plugin-reported). A new stats JSON-RPC method, polled only from runtimes that advertise capabilities: ["stats"] in initialize (old runners are simply not polled). In py-sdk, collect metrics with @app.stats: multiple collectors share a 4 s budget and merge into a flat numeric map, stored one row per metric in _stats_plugin_metrics. Both tables are queryable system collections; retention is 7 days.
  • API. GET /api/plugin-instances/:name/stats?window=3600 returns bucketed process and per-metric time series plus the latest report’s meta; GET /api/plugins/stats returns the latest snapshot of every instance with per-plugin rollups (instances, RSS/CPU/FD totals).
  • Console. The instance modal shows stat tiles (RSS, CPU %, FDs, uptime) with history charts and a custom-metric picker with a multi-line chart; the plugins page shows a totals strip per plugin.

Storage: the server can no longer lose its WAL files

Section titled “Storage: the server can no longer lose its WAL files”

Two failure paths could delete a live database’s -wal/-shm while pooled connections still held them open: a shared-memory reference-counting race in the vendored SQLite fork under connection churn, and any external SQLite client closing the same file cross-process. Connections left on deleted inodes read silently stale/empty data (an API list suddenly returns []) and fail writes with database disk image is malformed until restart; the on-disk database was never actually damaged, but the server was effectively poisoned.

All three layers are fixed:

  • The vendored fork never unlinks -wal/-shm: files left by a crash or external tool are recovered on the next open. (The fork’s deliberate close-on-reset behavior that cured the benchmarked fd leak is untouched; only the unlink calls are gone.)
  • Each database holds a permanent guard connection, so no in-process close path can ever believe it is the last connection.
  • Out-of-SQLite sentinel file locks (shared lock over the database file’s lock region and the shm dead-man byte), refreshed every second, make any external client’s last-connection cleanup see a live holder and stop deleting.

Regression tests cover the churn path and guard boundedness (internal/db/fdleak_test.go); the full suite is green.

  • Plugin instance configuration is now schema-driven forms: typed inputs, selects for enum fields, secrets with show/hide, JSON fallback, for both create and edit, instead of raw JSON textareas.
  • The plugin catalog moved into a modal behind the Scan button; the plugins page leads with stat cards and instances.
  • The sidebar can be hidden (choice remembered per browser), with a floating reveal button in the top-left corner.
  • Pipeline builder: combined Save + actions dropdown (Settings, Validate) and a truncated pipeline title with the full name on hover.
  • Consistent page headers and actions on Pipelines and Workflows (Refresh, then New), and the Authentication page uses the full content width.
  • Backup.md gains “Data directory access rules”: the server is the only process that should touch DATA_DIR; -wal/-shm are normal, never hand-pruned; replicas sync over HTTP and never read the master’s data directory; file-level backups require a stopped server or a replica.
  • Replication.md states the HTTP-sync model explicitly; Plugins.md and Pipelines.md document step-type declarations and the async step contract.