VSKI v0.12.0

Plugins: supervised Python beside the server (experimental)

Every real deployment grows a system around VSKI: an embeddings indexer, a record-to-graph sync, a crawler, a chat bot. Until now those lived as undocumented sidecars - cron entries, hand-rolled daemons, HTTP triggers pointing at localhost - with no shared lifecycle, no configuration story and no observability. v0.12.0 makes that pattern first-class: plugins are Python processes supervised by the server, configured and observed through the admin API and dashboard, running in the same container with zero network hops.

The design stance matters as much as the feature:

• Plugins are reusable source, not packages. Think shadcn components: you copy one into your distribution, verify its dependencies against the rest of your environment, tune its config schema, and build one image that just works. The server never installs or manages Python dependencies - the official images ship no Python at all, and distributions that want plugins bring their own locked interpreter (the wiki has the Dockerfile snippet).
• An instance is data. Plugin + database + JSON config + enabled flag. Changing an API key or cloning a bot for another chat group is a PATCH or a POST that restarts exactly one process - never the server, never the container.
• Not a sandbox. Plugin code ships inside your image and is trusted, exactly like workflow workers. There is no upload endpoint; code arrives via the filesystem only.

Plugins are experimental and disabled by default (PLUGINS_ENABLED=true to switch on; when off, every plugin endpoint answers 503 with setup instructions).

How it works

A plugin is a directory under PLUGINS_DIR (default DATA_DIR/plugins) containing a vski-plugin.json manifest (name, runtime entry, config schema with secret fields, declared deps as metadata) and a Python package. The supervisor spawns $PLUGIN_PYTHON -m vski.plugin.runner inside it - PLUGIN_PYTHON must have vski-sdk>=0.5.0 (and the plugin’s dependencies) installed by the distribution.

• Control plane: protocol frames on the process’s stdin/stdout (initialize / ping / invoke / shutdown), logs on stderr. A warm invocation is a pipe write - no socket, no cold start.
• Health: pinged every 10s; an unresponsive process is killed and restarted. Crashes restart with exponential backoff (1s doubling to 60s, reset after 5 minutes of uptime); the restarts counter tracks crash restarts. kill -9 recovery is covered by the e2e suite.
• Credentials: each process start mints a fresh API key with the plugin role and deletes it when the process exits - key material exists only in server memory and the child environment. The key is a normal authenticated principal: collection and graph access rules apply to it.
• Events: plugin handlers subscribe through the existing realtime websocket using a per-instance consumer group - at-least-once delivery with ACK-committed group offsets, so events that occur while a plugin is down are redelivered when it comes back (see the realtime section below).
• Replicas never supervise plugins (like webhooks and realtime); instance status stays readable through the synced registry.

API

All admin-only:

• GET /api/plugins, POST /api/plugins/scan - catalog from manifests
• GET/POST /api/plugin-instances (instances are unique per (dbName, name)), GET/PATCH/DELETE /api/plugin-instances/:name?db=
  • a config change bumps configVersion and restarts only that instance’s process
• POST .../:name/start|stop|restart
• POST .../:name/invoke - {function, params, timeoutSeconds?} (default 30s, max 120s) over the warm pipe
• GET .../:name/logs?level=&limit= - captured stderr (newest first; kept 7 days in _plugin_logs in stats.db)

Collection triggers and HTTP cron jobs accept plugin://<instance>/<function> as their URL: the webhook engine dispatches it in-process with the same retry policy and the same _webhook_logs pipeline - a zero-hop alternative to pointing a trigger at localhost. Cron jobs resolve their own database for the instance lookup.

Dashboard

Settings → Plugins: catalog cards from the manifests, the instances table with live status chips / restart counts / uptime, a config editor generated from each manifest’s schema (saving restarts just that process), an invoke console (function picker + params + result), a captured-log tail, start/stop/restart actions, and a disabled-state banner pointing at the env flags. The dashboard now depends on @vski/sdk@^2.2.0.

Writing a plugin

from vski.plugin import Plugin

app = Plugin()

@app.on_event("documents")      # realtime consumer group, at-least-once
async def on_document(ctx, event): ...

@app.function("reindex")        # callable via /invoke and plugin:// triggers
async def reindex(ctx, params): ...

@app.cron("*/5 * * * *")        # dependency-free 5-field cron
async def sweep(ctx): ...

ctx carries a db-scoped client, the instance config and a log whose output the server captures. Four working examples ship in examples/plugins/: hello (every handler type, doubles as the e2e fixture), note-bot (the wiki tutorial plugin), embeddings-indexer (event-driven chunking/embedding via an OpenAI-compatible endpoint - formalizes the Embeddings tutorial’s “next steps”) and graph-sync (mirrors a collection into a light graph with properties.ref links - formalizes the Graphs tutorial’s sync pattern).

Realtime: durable consumer groups and fixes

Work on plugin event delivery hardened the realtime gateway for everyone:

• API keys are now accepted on /api/realtime (?auth=<key>, any enabled key) - service processes can subscribe like any client.
• Consumer groups are durable. ACKs now advance a per-group committed offset, and consumers joining later (or plugin processes restarting) resume after it instead of tailing from “now” - events that fired while nobody was connected are no longer skipped. Group deliveries now carry the ackId field both SDKs already expected (neither ever ACKed before, because the server never sent it).
• Fixed: a fresh group consumer received one stale event from the head of the untrimmed event store (delivery position initialized from zero instead of the subscription position).
• Fixed: event frames carried the id of the first stored event with a matching action rather than their own, so ACKs could commit the wrong position.
• Event ids are zero-padded (<millis>-<seq>), making the id ordering used throughout the gateway numerically correct.

Non-group subscriptions are unchanged. Delivery remains at-least-once for groups.

Supervisor hardening

Start/stop transitions are serialized per instance (a restart can no longer race the reconciler into double-spawning an orphaned process), shutdown waits are bounded, and no process is spawned once shutdown has begun.

SDKs

• TypeScript/JS (@vski/sdk v2.2.0): new client.plugins namespace - list, scan, listInstances, getInstance, createInstance, updateInstance, deleteInstance, startInstance/stopInstance/restartInstance, invoke, logs - with full types (PluginCatalogItem, PluginInstance, PluginRuntimeInfo, PluginLogEntry).
• Python (vski-sdk v0.5.0): the vski.plugin module described above, a matching client.plugins management namespace, and three transport fixes: request paths with an embedded query string were silently dropped (httpx replaces a URL’s query when given params, even empty), websocket auth tokens are now URL-quoted (raw-base64 API keys containing + previously failed as spaces), and group subscriptions track lastId so a reconnecting consumer resumes after the last seen event.

Documentation & tests

Wiki: Plugins reference (architecture, manifest spec, env contract, API, security model, distribution Dockerfile) and a hands-on Plugins Tutorial whose flows are covered end-to-end by the new suites: 48_plugins_test.ts (15 steps: catalog, lifecycle, warm invoke, event delivery, captured logs, per-instance restart on config change, plugin:// trigger delivery, clone-into-second-db, crash recovery, admin gating, disabled-503) and test_48_plugins.py (instance lifecycle, event delivery, plugin:// triggers, admin gating, including downtime redelivery). The e2e harnesses now start their servers with plugins enabled and the examples directory mounted.

Compatibility

• No breaking API or storage changes. Plugins are off by default; the plugins / plugin_instances tables (default.db) and _plugin_logs (stats.db) are created idempotently on boot.
• Realtime group consumers start ACKing and resume from committed offsets - strictly fewer lost events; clients that manually ACKed are unaffected. The padded event-id format only matters within a single server process (the event store is in-memory).
• Official docker images are unchanged (no Python); plugin-enabled distributions build FROM them per the wiki.