Containers Plugin
The containers plugin adds a container step type to
pipelines: any Docker image runs as a short-lived,
resource-capped job, and files move in and out through the files API.
Think of each container as a cloud function with a Dockerfile. The plugin pulls or builds the image, starts the container with resource limits, enforces its timeout, captures its output, copies files in and out, removes it, and resumes the pipeline run. It is an official plugin; the source and the complete parameter reference live at vski.sh/x/containers.
What it does
Section titled “What it does”sequenceDiagram participant Run as pipeline run participant Plugin as containers plugin participant Files as files API participant Docker as host Docker daemon Run->>Plugin: start the step (params + run context) Plugin->>Files: download inputs Plugin->>Docker: pull or build the image, create the container Plugin->>Docker: copy inputs in, start, watch (limits, timeout) Docker-->>Plugin: exit code and logs Plugin->>Docker: copy artifacts out, remove the container Plugin->>Files: upload artifacts Plugin-->>Run: resume the step with the job summary- A native step type.
type: containersteps are validated against the plugin’s JSON Schema when a pipeline is saved, and the dashboard’s pipeline builder renders a form for them. The step output is the job summary: exit code, output tail, duration, peak CPU and memory, artifacts. - Any image, or your own Dockerfile. Run a published image as is, copy your scripts into it, or build a Dockerfile that is committed next to the pipeline’s IaC (or uploaded).
- Limits and isolation. Memory, CPU and process limits (clamped to the
instance’s maximums), a timeout, no network unless both the step and the
instance allow it, all Linux capabilities dropped,
no-new-privileges. - Files in and out.
inputscopies files into the container before it starts: an uploaded record’s file, an earlier step’s artifacts, a URL, or a script committed next to your IaC (file://scripts/report.py).artifactsuploads result files after a successful exit and lists them in the step output. - Logs. The last 64 KiB of stdout/stderr is the step’s
output; the full log stays on the data volume. - Crash recovery. Containers outlive plugin and server restarts: the plugin re-attaches to running ones, finishes those that exited in the meantime, resumes interrupted uploads and resumes the parked runs.
- Cleanup and stats. A container is removed as soon as its exit is recorded, an hourly sweep removes leftovers and expired artifacts, and running jobs, CPU and memory show on the dashboard’s Plugins page.
Use cases
Section titled “Use cases”- File processing on upload. A collection trigger fires when a file is uploaded; the container receives it (pandoc, LibreOffice, ImageMagick, ffmpeg, an OCR engine, …) and the results come back as downloadable artifacts. See the example below.
- Scheduled scrapers and API pollers. A cron-triggered pipeline runs the scraper with networking opted in; its output or a JSON artifact feeds the next step.
- Data jobs and reports. pandas or DuckDB transforms, exports and reports that read their inputs and publish CSV, Parquet or HTML files.
- Builds and tests.
make testor a linter in a pinned toolchain image; a nonzero exit fails the run. - Code written by agents. An
llmstep or a coding agent (through vski mcp) writes a script, and a container step runs it in a throwaway, network-isolated container instead of inside the server. This isolates semi-trusted code; it is not a sandbox for hostile code (see Security). - Tools outside Python. CLIs such as gdal, ffmpeg or pandoc run in their own images instead of being installed into the vski image.
Install
Section titled “Install”Requirements
Section titled “Requirements”- A Linux host with Docker. The plugin starts sibling containers through the
host daemon’s socket,
/var/run/docker.sock, so step images must be pullable by that daemon (or pre-pulled). - vski ≥ 0.17.5 with plugins enabled.
- A plugin interpreter (
PLUGIN_PYTHON, Python ≥ 3.11) withvski-sdk0.15 and the plugin’srequirements.txt(docker,httpx). - To apply IaC that references files with
file://: the vski CLI from vski-sdk ≥ 0.17.
Option 1: the bundled distribution
Section titled “Option 1: the bundled distribution”The plugin repository builds a ready-to-run image: the standalone vski image plus Python, the plugin and its dependencies, with plugins enabled.
git clone https://vski.sh/x/containers.git && cd containerscp .env.example .env # set JWT_SECRET (openssl rand -hex 32)make up # build and start on SERVER_PORT (default 3000)Open the dashboard and create the first admin. docker-compose.yml mounts
the data volume and the Docker socket:
volumes: - ./data:/app/data - /var/run/docker.sock:/var/run/docker.sockOption 2: add it to your own distribution
Section titled “Option 2: add it to your own distribution”Plugins are source components that ship inside your image. Put the plugin
repository in your build context as containers/, install its
dependencies into the plugin interpreter, and copy it into the plugins
directory:
# or start from your existing vski imageFROM vski.sh/x/vski:latest-standalone
RUN apt-get update && apt-get install -y --no-install-recommends python3 python3-pip \ && rm -rf /var/lib/apt/lists/*COPY containers/requirements.txt /tmp/containers-requirements.txtRUN python3 -m pip install --break-system-packages --no-cache-dir \ --extra-index-url https://vski.sh/api/packages/x/pypi/simple/ \ -r /tmp/containers-requirements.txtCOPY containers/ /app/plugins/containers/
ENV PLUGINS_ENABLED=true PLUGINS_DIR=/app/plugins PLUGIN_PYTHON=python3Run the image with the Docker socket mounted
(-v /var/run/docker.sock:/var/run/docker.sock). Keep PLUGINS_DIR outside
the data volume: a volume mounted at /app/data hides anything the image
put there.
Create an instance
Section titled “Create an instance”A plugin runs once it has an instance in a database. Create one on the dashboard’s Plugins page, through the plugin API, or with IaC:
version: 1database: default
plugins: containers: plugin: containers enabled: true config: max_concurrent: 2 max_memory: 1g allow_network: true # let steps opt into networking (off by default)The instance config caps what steps can request (see
Configuration). Pipelines in that database can now use
type: container.
Check that it works
Section titled “Check that it works”version: 1
pipelines: hello-container: enabled: true steps: - id: hello type: container timeout: 60 params: image: alpine:3.20 command: echo "hello from Alpine $(cat /etc/alpine-release)" memory: 64m output: exitCode: "{{steps.hello.output.exitCode}}" output: "{{steps.hello.output.output}}"vski iac apply # the instance and the pipelinevski pipelines run hello-container # prints the run idvski runs show <run id> # output: exitCode 0, "hello from Alpine 3.20.x"The first run pulls alpine:3.20; later runs reuse it. The plugin
repository’s vski/ directory has more examples (make iac-apply applies
them to the local stack).
Writing container steps
Section titled “Writing container steps”A step names an image (or a Dockerfile) and a command, plus optional limits, environment and files. Scripts are files of their own, committed next to the IaC that uses them:
- id: scrape type: container timeout: 600 # the step budget, and the container's lifetime params: image: python:3.12-slim inputs: file://scripts/scrape.py # copied in as scrape.py command: python scrape.py env: API_TOKEN: "{{secrets.api_token}}" # resolved by the server, never stored in the definition network: true # needs allow_network on the instance memory: 256m cpus: 1import urllib.request
print(urllib.request.urlopen("https://example.com").status)A string command runs through /bin/sh -c; a list runs as is. The step
fails on a nonzero exit, a timeout, a stop or a failed file transfer, with
the output tail in its error. {{steps.scrape.output.output}} is the output
tail and {{steps.scrape.output.exitCode}} the exit code. Unknown
parameters are rejected, which is how volumes, devices, privileged mode and
host networking stay out. The README lists every parameter and the full
output shape.
Where files come from
Section titled “Where files come from”inputs, the build context files and dockerfile all take files from the
same sources:
| Source | Written as |
|---|---|
| A file committed with your IaC | file://scripts/report.py, relative to the YAML file |
| An uploaded file | /files/<collection>/<recordId>/<filename>, or a reference {collection, recordId, filename}: a record’s file field, a trigger’s record, an earlier step’s artifact |
| A URL | https://…, when the instance sets allow_external_inputs |
| Inline text | the string itself, in files and dockerfile only |
file://references are resolved byvski apply, which embeds each file into the pipeline definition as adata:URL (at most 256 KiB per file). The file reaches the container byte for byte:{{…}}inside a script stays as written, so values reach it throughenv. A definition created any other way cannot usefile://: upload the file and reference its/files/…path instead.- Inline text is a pipeline string: templates expand in it, and stray
{{or}}(a JSON literal, a Jinja template) fail validation. Keep it to a few lines of shell. - Large files (datasets, models, archives) go through the files API, not into the definition.
- In
filesanddockerfile, a string is inline text unless it is afile://reference; write an upload or a URL as{from: <source>}:
- id: convert type: container params: dockerfile: file://images/convert/Dockerfile files: # the build context convert.py: file://images/convert/convert.py requirements.txt: {from: /files/build_files/k3j2…/requirements.txt} run.sh: | # a few lines inline are fine #!/bin/sh exec python /app/convert.py "$@"Files in and out
Section titled “Files in and out”Files never travel through the pipeline itself: steps pass small references to files stored by the files API (see Pipelines → Files between steps).
- id: merge type: container dependsOn: [fetch] params: image: alpine:3.20 workdir: /work inputs: "{{steps.fetch.output.artifacts}}" # each file at its name, e.g. out/a.csv command: mkdir -p result && cat out/*.csv > result/merged.csv artifacts: result/merged.csv # uploaded after exit 0-
inputstakes any of the sources above, or{from, dest, optional};frommay be a whole list. Files are fetched (size and checksum verified when the reference carries them) and copied into the container before it starts. An upload that is not stored yet is retried for 30 seconds, so pipelines can trigger on a record’screateevent. -
artifactstakes paths or globs (out/*.json,out/**, a whole directory), relative to the workdir. After a successful exit the matching files are copied out and uploaded, one record per file, to thepipeline_artifactscollection (created on first use), andsteps.<id>.output.artifactslists them:{ "kind": "file", "name": "result/merged.csv", "filename": "merged.csv","size": 1234, "sha256": "…", "collection": "pipeline_artifacts","recordId": "…", "path": "/files/pipeline_artifacts/…/merged.csv", … }Download any artifact with
GET <server><path>.
Example: reports from uploaded CSV files
Section titled “Example: reports from uploaded CSV files”Every CSV uploaded to a datasets collection becomes an HTML report. The
report script is a file next to the IaC that declares the collection and the
pipeline:
vski/├── default/collections.yaml├── pipelines.yaml└── scripts/report.pyversion: 1database: default
collections: datasets: type: base fields: - {name: title, type: text} - {name: data, type: file, required: true}version: 1
pipelines: dataset-report: enabled: true triggers: - {type: collection, collection: datasets, actions: [create]} steps: - id: render type: container timeout: 120 params: image: python:3.12-alpine workdir: /work inputs: - file://scripts/report.py # lands at report.py - from: # the uploaded CSV collection: "{{trigger.collection}}" recordId: "{{trigger.id}}" filename: "{{trigger.record.data}}" dest: in/data.csv env: TITLE: "{{trigger.record.title}}" # empty when the record has no title command: python report.py artifacts: out/report.html memory: 128m output: report: "{{steps.render.output.artifacts[0].path}}""""Render in/data.csv as an HTML table in out/report.html."""
import csvimport htmlimport osimport pathlib
with open("in/data.csv", newline="", encoding="utf-8") as handle: rows = list(csv.reader(handle))if not rows: raise SystemExit("in/data.csv is empty")head, body = rows[0], rows[1:]
def cell(tag: str, value: str) -> str: return f"<{tag}>{html.escape(value)}</{tag}>"
table = "".join( "<tr>" + "".join(cell(tag, value) for value in row) + "</tr>" for tag, row in [("th", head)] + [("td", row) for row in body])title = html.escape(os.environ.get("TITLE") or "report")pathlib.Path("out").mkdir(exist_ok=True)pathlib.Path("out/report.html").write_text( f"<h1>{title}: {len(body)} rows</h1><table>{table}</table>\n", encoding="utf-8")print(f"rendered {len(body)} rows")vski apply creates the collection and the pipeline, with the script
embedded in it; editing report.py and applying again updates the pipeline.
Upload a file, here with the Python SDK:
with open("q3.csv", "rb") as f: await client.collection("datasets").create( {"title": "Q3"}, files={"data": ("q3.csv", f, "text/csv")} )The upload starts a run of dataset-report. When it completes, its output
holds the report’s path, e.g.
/files/pipeline_artifacts/3e11…/report.html; open it on your server. The
report’s record in pipeline_artifacts carries the run and step ids, so
every report can be traced back to the run that produced it. The plugin
repository ships this example as container-dataset-report.
Configuration
Section titled “Configuration”The instance config caps what steps can request (requests above a maximum are clamped, with a warning in the step output):
| Key | Default | Meaning |
|---|---|---|
max_concurrent | 2 | Containers (and builds) running at once; further jobs queue |
default_memory / max_memory | 256m / 1g | Memory limit when a step sets none / the hard cap |
default_cpus / max_cpus | 1.0 / 2.0 | CPU limit default / cap |
default_timeout / max_timeout | 600 / 3600 | Container lifetime in seconds, default / cap |
allow_network | false | Steps may opt into networking (network: true) |
allow_build | true | Steps may build images (dockerfile, or image + files) |
artifacts_collection | pipeline_artifacts | Collection that stores artifacts |
max_input_size / max_artifact_size | 512m / 1g | Total size of the files a step fetches (inputs and build files) / of its artifacts |
artifact_retention_hours | 720 | Age at which artifacts are deleted; 0 keeps them |
allow_external_inputs | false | Allow http(s) URLs as inputs or build files (fetched by the plugin process) |
The plugin README documents the rest (pids limits, pull policy, log and job retention, transfer timeouts).
Operations
Section titled “Operations”- Logs. A step’s
outputis its stdout/stderr tail. The plugin functioncontainer.logsreturns the full log (it also stays on the data volume undercontainer-logs/), and the plugin’s own log is on the dashboard’s Plugins page. - Functions.
container.status,container.stop,container.logs,container.artifactsandcontainer.validate(a dry run that validates step params) can be invoked from the dashboard or withPOST /api/plugin-instances/<instance>/invoke. - Stats. The Plugins page charts
jobs.running,jobs.queued,jobs.collecting,containers.cpu_pctandcontainers.mem_mb, plus per-container CPU and memory, sampled every 10 seconds. - Restarts. Restarting the plugin, the server or the whole container never loses a job: running containers are re-attached, and runs parked on a container step resume when it exits.
Security
Section titled “Security”- The Docker socket is root-equivalent on the host. Anyone who can
create pipelines (admins and
workflows-role principals) can run any image through the host’s Docker daemon. Use the plugin for trusted pipelines (personal, lab, internal automation), never for untrusted tenants. - Containers run without network by default (the step and the instance must
both opt in), with all capabilities dropped,
no-new-privileges, and memory, CPU, process and time limits. Volumes, devices, privileged mode and host networking are rejected; files move through Docker’s archive API, never through mounts. - URL sources are off by default: the plugin process fetches them, so enabling them lets pipeline authors reach anything the server can reach.
file://references are read by thevskiCLI on the machine that applies the IaC, and only inside the IaC directory (symbolic links included), so a pipeline file cannot pull in~/.sshor a CI runner’s secrets.- Files, artifacts included, are served by a files route that does not check credentials: anyone who can reach the server and knows a file’s path can download it. Keep secrets out of artifacts.
Limitations
Section titled “Limitations”- Linux hosts only.
- Artifacts are collected only from containers that exit 0; a failed step keeps its full log, not its files.
readOnlycontainers cannot take inputs.- Files embedded with
file://are limited to 256 KiB each and 1 MiB per pipeline: they live in the definition and in every run’s copy of it. - Completion is reported at least once: a crash at the wrong moment can deliver a duplicate completion signal, which arrives after the step has completed and sits unused.
See also
Section titled “See also”- Plugins: how plugins are discovered, configured and supervised.
- Pipelines: triggers, templates and files between steps.
- Tutorial: Plugins: write a plugin of your own.
- vski.sh/x/containers: source, the full parameter reference and the example pipelines.