VSKI v0.17.7
Replication is rebuilt around a durable, verifiable changeset log: every committed write is appended to an on-disk log before its transaction commits, replicas resume from an exact position, and a promoted replica serves its former siblings from the same log. Nodes switch between master and replica in-process, without a restart. On top of that ships tango, a new cluster proxy: it splits reads across replicas, journals every write to Redis, and fails over to the most-current replica. Acknowledged writes survive the failover, and none is applied twice.
| failover under load (200 writes/s + 400 reads/s, master stopped) | |
|---|---|
| acknowledged writes missing on the new master | 0 |
| duplicated writes | 0 |
| write outage (503 + Retry-After) | 3.4 s |
| old master rejoined as replica | 2.0 s |
Upgrade notes:
- Upgrade replicas before the master. A master on the new protocol
answers older replicas with
426 Upgrade Required; they stay readable but stale until upgraded. Each replica re-downloads once after the upgrade. workflow_eventsgains aseqcolumn, added automatically on start.- Behavior changes, all listed below: replicas refuse more writes, the SQL console refuses transaction control, and transient write failures now return 503 instead of 403.
- No SDK changes.
Durable changeset log
Section titled “Durable changeset log”- Writes through the write queue are captured on the database’s pinned
connection and appended to
DATA_DIR/replication-log/<db>.clog(CRC-checked, fsynced) before their batch commits. A failed commit removes its entry; crash recovery drops uncommitted tails. - Entries form a hash chain. A replica resumes with its last position plus the chain hash at that position. A position that is pruned, ahead of the master, or on a different history (a failover onto a node that missed writes) gets a full download instead of a mixed history.
- Replicas mirror the master’s log entry for entry, so after a promotion the remaining replicas catch up incrementally from the new master. A master restart costs replicas nothing.
- Entries are kept for
REPLICATION_LOG_RETENTION_HOURS(24); a replica offline longer re-downloads once. A database no replica has asked for is not logged, so standalone nodes pay nothing. - Full downloads use SQLite’s backup API: snapshots include writes still in the WAL, and replicas restore in place, so open handles stay valid. Previously a replica’s first full download could leave services on a closed database.
- Writes from the SQL console and from migration scripts now go through
the write queue, so they replicate. Inside them, transaction control
(
BEGIN,COMMIT,ROLLBACK, savepoints),ATTACH/DETACHand pragma assignments are refused: each request already runs in its own transaction. - A changeset carrying columns or tables a replica lacks is no longer silently skipped: the replica syncs the schema first.
- Workflow events carry a per-run
seqassigned when they commit, so a promoted replica hands workers the same event order as the old master.
Roles, failover and replica safety
Section titled “Roles, failover and replica safety”- Hot promote/demote:
POST /api/admin/promoteandPOST /api/admin/demote(superuser token) switch a node’s role in-process: write path, syncer, changeset journal, pipeline dispatcher, cron, workflow gateway, webhooks, graph index. - Workflow execution is master-only: workflow gateway, scheduler,
cron dispatch, webhooks and realtime run on the master alone.
Replicas answer workflow writes and queue polls with
503. workflows.dbreplicates by default as a passive copy, so a promoted replica starts with every run, queued job, timer and definition. Jobs the dead master had handed out are reclaimed within 30–60 s and redelivered.REPLICA_INCLUDE_WORKFLOWS=falsekeeps it node-local.- Replicas no longer write locally. Every write to a replicated
database is refused below the HTTP layer, so a replica never holds rows
its master never saw. Logins still work on replicas. Registration,
OAuth account linking, admin init and API key creation now return
403there and must go to the master; before, they silently created diverging rows. - Exactly-once replay: a write carrying
X-Tango-Idempotency-Keyrecords the key in the same transaction; a repeat answers200 {"duplicate":true}without writing. Keys replicate with the data and are pruned afterIDEMPOTENCY_KEY_RETENTION(2h), which must exceed tango’s journal retention. - Older replicas (pre-log protocol) get
426 Upgrade Required, logged once per replica every few minutes. - Security:
/api/admin/replicas*now requires an admin. Previously any authenticated user could mint replica tokens. - Replicas report their positions to the master after every sync
(
POST /api/replica/sync-complete); every node exposes its own atGET /api/replica/local-status.
Write path fixes
Section titled “Write path fixes”- Transient failures are retryable: a write hitting a node that is
shutting down, a full write queue, a replica, or a busy database now
returns
503withRetry-Afterinstead of403, which clients treated as final. - Write-queue batches take the write lock up front (
BEGIN IMMEDIATE). A batch that read before writing could fail with “database is locked” when another connection committed in between; SQLite reports that without waiting. - Deleting a database released neither its replication session nor its connection handle, which could hang shutdown or crash the server. Sessions are now released before their connection closes.
- The Docker images are built with SQLite sessions enabled. Without them a replica’s first snapshot failed, so containers could not replicate.
- The Docker images are published for linux/amd64 and linux/arm64.
tango: the cluster proxy
Section titled “tango: the cluster proxy”A new component (tango/): one endpoint in front of a vski master and
its replicas.
- Routing: writes go to the master. Reads go to the least-busy
healthy node, except the workflow and automation plane (workflows,
pipelines, cron, webhooks, plugins, secrets) and the realtime and
workflow websockets, which stay on the master. Sessions (
/sso, logins,/api/auth,/api/admins,/api/keys) and admin scripts (SQL console, migrations) go to the master and are never journaled. - Write journal: every write is appended to a Redis Stream before it is forwarded, with an idempotency key the master stores with the write. On failover tango replays, in order, every entry the new master may lack, including writes the old master acknowledged but never replicated. Writes whose client saw a failure are dropped, never replayed.
- Failover: tango fences the old master, hot-promotes the replica
with the most entries applied, then replays. In
strictmode (default) writes get503+Retry-Afterduring the window; inabsorbmode they are journaled and answered202. - Read freshness: a replica that falls behind the master by more
than
TANGO_MAX_READ_LAG(30s) leaves the read pool until it catches up. Optional read-your-writes (TANGO_READ_YOUR_WRITES) pins a client’s reads to the master briefly after its write. - Actuators carry out role changes:
docker: Engine API, role files read by containers at start; replicas sync through tango, so promotions need no reconfigurationkubernetes(tango v1.0.2 or later): a role ConfigMap with one entry per pod and lease election between tango replicas; replicas sync through tango’s management Service, as with docker. The vski-cluster Helm chart installs the stacklocal: child processes, for developmentmanual
- Management API (
:8001): status, manual promotion, replay.
Measured (tango/bench, three nodes on one host, k6 open model),
p95 added by tango:
| 500/s | 1,000/s | 2,000/s | 4,000/s | |
|---|---|---|---|---|
| reads (point, list) | +0.2 ms | +0.2–0.3 ms | +0.1–0.2 ms | +0.1 ms |
| record creates | +0.1 ms | +0.1 ms | +5.7 ms | +8.0 ms |
No errors at any rate. On writes, most of the cost is the idempotency key vski stores with each write; the benchmark breaks the cost down.
Tests and docs
Section titled “Tests and docs”- New end-to-end suites cover multi-replica convergence, master restarts, hot promotion and switchover, incremental catch-up after promotion, workflow state across failover, and divergence handling. vski’s full end-to-end suite (47 tests) passes through tango, and tango’s failover test proves acknowledged-but-unreplicated writes survive exactly once.
- Replication.md documents the changeset log, upgrade order, replica read-only rules and the new settings; tango ships its own README and benchmark guide.