Skip to content

Knowledge, Workflow, Human In The Loop

Classic software architecture separates concerns into layers with defined contracts between them: MVC splits an application into model, view and controller, and network stacks separate transport from session from application. Agentic systems deserve the same discipline. The literature proposes several layerings, usually along planning, retrieval, tool use and memory, with definitions that vary from paper to paper. We prefer to separate three abstract layers: (K) Knowledge, (W) Workflow and (H) Human In The Loop. Every agent carries all three, from a one-shot script to a multi-agent fleet. The question is never whether the layers exist, only whether each one was designed or left accidental.

The layers are a decomposition for building and operating agents, not a taxonomy for arguing about them. Each layer owns one kind of failure, is evaluated on its own scorecard, and maps to concrete platform primitives, so the architecture lands as configuration rather than as a diagram.

LayerOwnsThe question it answersSignature failure
K, Knowledgeeverything the agent knows: retrieval, analysis, synthesisIs it true, and can a human verify it?confident wrong facts
W, Workfloweverything the agent does: procedures, tools, budgetsIs it the right action, within limits, on the record?wrong or runaway actions
H, Human In The Loopeverything that connects the system to accountable people: gates, escalation, reviewCan a human see it, understand it, and stop it?rubber-stamped gates, surprised humans

The K layer is everything the agent knows: knowledge retrieval, analysis, synthesis, reporting. The agent pulls information and produces a structured output: findings, briefs, bundles. That output is a contract, not prose: named fields, evidence behind every claim, a confidence number, sources a reader can open.

Four properties define a healthy K layer:

  1. Retrieval correctness. The right rows and documents, not the nearest ones.
  2. Scope discipline. The question that was asked, not the neighborhood of it.
  3. Evidence. Every claim traceable to something a human can open.
  4. Output contract. Same fields, same shape, every time. Drift is a defect, not a style change.

Knowledge work looks harmless, which is why it rots first: a report with a wrong number in it is not a writing bug, it is a missing eval. The practice is to evaluate the whole trajectory (what was pulled, in what order), each step, and the final output against a golden dataset, and to turn every production failure into a new test case.

K concernPlatform primitiveWhy it holds
Output contractCollections with typed, required fieldsthe schema is the contract; validation is server-side, so a malformed brief cannot be written at all
Scope disciplineRuleslist and view rules are SQL expressions applied as query filters; the caller never retrieves what the rule excludes, so scope is enforced at storage, not in the prompt
Keyword retrievalFull-Text Search (FTS5)automatic on text collections; no separate search service
Semantic retrievalVector Embeddings (sqlite-vec)vectors live beside the records in the same database; hybrid keyword plus vector queries in one path
Multi-hop evidenceGraphstyped edges between records; traversal answers relationship questions without prompt-side reasoning
Shaping and joinsViewsSQL views expose exactly the slice each consumer needs, computed where the data lives
Probe setscollections againgolden datasets and failure-derived test cases are just records; evals query them like any other corpus

The W layer is where the agent stops describing and starts acting. Every actionable item the system can perform is a named procedure with preconditions, parameters, guards, a budget and a journal entry. If the system can restart a service, that is a W procedure. If it can spend money, that is a W procedure with a gate in front of it. The procedures are the substance of the agent; everything else feeds them or audits them.

Four properties define a healthy W layer:

  1. Least-privilege tool allowlists. The agent gets exactly the tools its procedures need, enforced at the server layer, never in the prompt. Every extra tool widens what a prompt injection can reach.
  2. Budgets. Hard ceilings on retries, timeouts, concurrency and wall clock, per step and per pipeline. A loop that cannot stop is an outage with a progress bar.
  3. Scope refusals. Anything outside declared scope is refused by rule, and the refusal is an event somebody sees.
  4. Degradation. An autonomous action that cannot proceed safely degrades to a human decision instead of failing silently or acting outside its scope. Degrading is a designed outcome, not a failure.
W concernPlatform primitiveNotes
Durable proceduresWorkflowssteps are journaled server-side; a crash or deploy resumes the run where it stopped, instead of replaying side effects
Reactions to dataPipelinescollection triggers (create, update, delete) start runs the moment a record lands
Time-driven procedurespipeline cron triggers and the Cron tableschedules are part of the pipeline definition; cron jobs can also kick pipelines ad hoc with pipeline://
External entry pointspublic trigger tokensunauthenticated POST /api/pipelines/trigger/:token, rate limited, revocable; how CI and third parties start runs
Tool allowlistPluginsplugin steps can only call declared plugin instances over the in-process pipe; there is no tool surface beyond what is declared
Budgetsper-step and per-pipeline optionsstep: retries, timeout, condition, onError; pipeline: concurrency, maxEvents, executionTimeout
Compositionstep graphdependsOn ordering, foreach fanout, sub-pipelines, and workflow steps with wait: true
Credentialssecrets (see Pipelines)encrypted at rest; values arrive from environment variables at apply time and never live in YAML

The H layer is the interface to the only component that can be held accountable: people. Human review, triage, escalation and ambient triggers live here, and so do the properties that make the other two layers safe to run in production.

Four properties define a healthy H layer:

  1. Gates in front of consequential actions. What the approver sees is the K layer’s brief plus the W layer’s blast radius, not a button with a label.
  2. Silence is not approval. A timeout escalates. “Nobody answered” is a state transition with an owner, never a default yes.
  3. Outputs shaped for people. Readable reports with confidence and evidence, not JSON dumps.
  4. Refusals explain themselves. “I will not roll back the database, because the evidence points at the deploy, not the schema” is an acceptable output; a bare error is not.
H concernPlatform primitiveNotes
Approval gatesworkflow signalsctx.wait_for_signal("manager-approval") suspends the run durably; client.workflow.send_signal(run_id, ...) resumes it with the decision
One-click review endpointsworkflow hookstoken-gated; executing the hook delivers its payload to the run as the signal, with the hook’s metadata merged in
Escalation timersctx.sleep in the workflow, or a cron sweepexpiry is journaled data, so “ten minutes of silence” can escalate to a lead deterministically
Hard gates on dataRulesa missing rule makes an operation admin-only; granular rules decide which fields a reviewer may touch
Triage surfaceRealtime subscriptions and the web UIgate requests, refusals and reports are records; subscribe to them like any other collection
Auditrun journal and workflow eventswho approved what, when, on which evidence; queryable after the fact

A deliberately small example: one store, one on-call engineer, one agent on a local model. Checkout errors cross 4.1% against a 1% floor at 03:07. The run below touches every layer, and every step leaves an artifact the next step reads.

#StepLayerOn the platformArtifact
1Ambient trigger: error rate crosses the floorHa cron-triggered pipeline compares the rate against the threshold and writes an alertalert record
2Evidence run: recent deploys, error logs, runbook, dependency statusKfull-text and vector queries over the deploy, log and runbook collectionsevidence records
3Incident brief: what changed, what broke, candidates, each with evidence and confidenceKwritten to a briefs collection whose schema is the output contractbrief record
4Procedure select: restart the checkout service, budget one restartWa workflow step picks from its allowlist; the non-consequential branch runsjournal entry
5The gate, had rollback been chosenHthe run suspends on an approval signal; a hook lets the on-call approve from chat; a ten minute timer escalates to the leadsignal payload
6Execute and journalWthe restart runs inside the allowlist, budget spent one of one, every step journaledrun events
7Postmortem bundle: what happened, what it costKone new eval case written to the probe seteval record
8Readable report: confidence, evidence, triage notesHa report record pushed to the on-call over realtimereport record

Note the refusal inside step 4: a database rollback is declined because the evidence points at the deploy, not the schema. That refusal is W layer work doing H layer work in advance: it explains itself before any human had to ask. No step skipped a layer, and no step needed a diagram to see it.

The layers give you the fault domain, which is most of debugging.

SymptomLayerFirst places to look
Wrong or missing factKthe retrieval query, rule scope, probe set coverage for this case
Wrong action, blown budget, silent failureWthe run journal, step conditions, retries and timeouts
Human surprised, rubber-stamped gate, ignored reportHwhat the gate displayed, escalation timers, where reports land
Looks like two layers at oncethe contract between themW reads fields K stopped writing; W journaled evidence the gate never displayed

Never one score. Each layer gets its own evaluation, run in CI on every change to any layer.

LayerWhat to measureWhere it lives
Kretrieval correctness on the probe set, citation coverage, contract conformanceprobe sets are collections; the eval is a query
Wdry-run trajectories against golden sequences, budget conformance, injection red teamthe run journal, replayed against a twin database
Htriage time (a teammate who has never seen the agent can triage an output in two minutes), approval latency, time-to-degradetimestamps in workflow events and run journals

Every layer’s production failures become next week’s eval cases. That loop is the difference between an agent that improves and an agent that decays.

All three layers land as Infrastructure as Code and apply with vski plan and vski apply, so the architecture is reviewable as a diff:

FileLayerWhat it declares
vski/<database>/collections.yamlKoutput contracts, rules, FTS fields, views, indexes
vski/pipelines.yamlWtriggers, steps, budgets, conditions, plugin targets
vski/cron.yamlW and Hschedules, pipeline kicks, escalation sweeps
vski/secrets.yamlWwhich environment variables deliver which credentials

The long version of this argument, with sources and the incident in full, is on the blog: The KWH Framework. Next stops in this wiki: Workflows for the durable engine, Pipelines for triggers and steps, and Collections with Rules for the K layer’s contracts.