VSKI v0.17.5
The plugin platform grows up in two directions: plugins can now
define their own pipeline step types (a container step instead of
the start → signal → check workaround) and report live stats; both
process metrics the server collects itself and custom metrics the
plugin publishes. Under the hood, the storage layer fixes a
connection-pool race that could make pooled connections read through
deleted WAL files (empty/stale reads, then database disk image is malformed on writes); that class of failure is now structurally
impossible from inside the server and blocked from outside it.
Safe upgrade: no schema migrations; stats tables are created in
stats.db on start and plugin catalogs are re-scanned automatically.
Everything is on by default. SDKs: @vski/sdk 2.10.0, vski-sdk
0.15.0.
Plugin-defined pipeline step types
Section titled “Plugin-defined pipeline step types”A plugin manifest can now declare step types alongside its functions:
steps: container: description: Build/run an ephemeral Docker container and report its exit code and output. params: containers/step.schema.json # file path or inline JSON Schema functions: start: container.start # required; must ack fast check: container.check # optional, informational- Registry and validation. Scanning inlines the params schema into
the catalog; step types resolve per database from plugins with an
enabled instance. Type names must match
^[a-zA-Z0-9_-]{1,64}$and collisions across plugins are rejected at validation. Param templates ({{...}}) are checked as usual; fully-resolved params are validated against the plugin’s JSON Schema (draft 2020-12). - Asynchronous execution. The server dispatches the start function
with the step params plus an injected
context(runId,stepId,pipeline,db,signal,timeout,attempt) under a 30-second dispatch budget: the function acks and does its work in the background. The step parks on a deterministic per-step signal; the plugin completes it withclient.workflow.send_signal(runId, signal, {ok, error, output}):ok: falsefails the step witherror, anything else completes it withoutput. Completion is durable: signals that arrive before the park are buffered and replayed,runId/stepIdgive the plugin an idempotency key for re-dispatches, and the steptimeout(1–86400 s) is passed through as the overall budget. Retries cover dispatch failures; a plugin-reported failure is terminal. - Builder. Plugin step types appear in the palette next to the builtins with their own icon and accent color, and the inspector renders a form from the params schema (typed inputs, enums as selects, JSON fallback for complex shapes).
- The containers plugin ships a native
containerstep: build/run an ephemeral container with resource caps, log capture and timeout enforcement, replacing the three-step workaround in a single node.
Plugin stats: process and custom
Section titled “Plugin stats: process and custom”Every plugin instance gets a live stats view, from two sources merged by the 30-second stats collector:
- Process stats (server-collected). Read from
/procper plugin child process: RSS, CPU % (utime+stime delta), open file descriptors, uptime, stored in_stats_plugin_processes. - Custom metrics (plugin-reported). A new
statsJSON-RPC method, polled only from runtimes that advertisecapabilities: ["stats"]ininitialize(old runners are simply not polled). In py-sdk, collect metrics with@app.stats: multiple collectors share a 4 s budget and merge into a flat numeric map, stored one row per metric in_stats_plugin_metrics. Both tables are queryable system collections; retention is 7 days. - API.
GET /api/plugin-instances/:name/stats?window=3600returns bucketed process and per-metric time series plus the latest report’s meta;GET /api/plugins/statsreturns the latest snapshot of every instance with per-plugin rollups (instances, RSS/CPU/FD totals). - Console. The instance modal shows stat tiles (RSS, CPU %, FDs, uptime) with history charts and a custom-metric picker with a multi-line chart; the plugins page shows a totals strip per plugin.
Storage: the server can no longer lose its WAL files
Section titled “Storage: the server can no longer lose its WAL files”Two failure paths could delete a live database’s -wal/-shm while
pooled connections still held them open: a shared-memory
reference-counting race in the vendored SQLite fork under connection
churn, and any external SQLite client closing the same file
cross-process. Connections left on deleted inodes read silently
stale/empty data (an API list suddenly returns []) and fail writes
with database disk image is malformed until restart; the on-disk
database was never actually damaged, but the server was effectively
poisoned.
All three layers are fixed:
- The vendored fork never unlinks
-wal/-shm: files left by a crash or external tool are recovered on the next open. (The fork’s deliberate close-on-reset behavior that cured the benchmarked fd leak is untouched; only the unlink calls are gone.) - Each database holds a permanent guard connection, so no in-process close path can ever believe it is the last connection.
- Out-of-SQLite sentinel file locks (shared lock over the database file’s lock region and the shm dead-man byte), refreshed every second, make any external client’s last-connection cleanup see a live holder and stop deleting.
Regression tests cover the churn path and guard boundedness
(internal/db/fdleak_test.go); the full suite is green.
Console polish
Section titled “Console polish”- Plugin instance configuration is now schema-driven forms: typed inputs, selects for enum fields, secrets with show/hide, JSON fallback, for both create and edit, instead of raw JSON textareas.
- The plugin catalog moved into a modal behind the Scan button; the plugins page leads with stat cards and instances.
- The sidebar can be hidden (choice remembered per browser), with a floating reveal button in the top-left corner.
- Pipeline builder: combined Save + actions dropdown (Settings, Validate) and a truncated pipeline title with the full name on hover.
- Consistent page headers and actions on Pipelines and Workflows (Refresh, then New), and the Authentication page uses the full content width.
Documentation
Section titled “Documentation”- Backup.md gains “Data directory access rules”: the server is the
only process that should touch
DATA_DIR;-wal/-shmare normal, never hand-pruned; replicas sync over HTTP and never read the master’s data directory; file-level backups require a stopped server or a replica. - Replication.md states the HTTP-sync model explicitly; Plugins.md and Pipelines.md document step-type declarations and the async step contract.