Skip to content

VSKI v0.17.7

Replication is rebuilt around a durable, verifiable changeset log: every committed write is appended to an on-disk log before its transaction commits, replicas resume from an exact position, and a promoted replica serves its former siblings from the same log. Nodes switch between master and replica in-process, without a restart. On top of that ships tango, a new cluster proxy: it splits reads across replicas, journals every write to Redis, and fails over to the most-current replica. Acknowledged writes survive the failover, and none is applied twice.

failover under load (200 writes/s + 400 reads/s, master stopped)
acknowledged writes missing on the new master0
duplicated writes0
write outage (503 + Retry-After)3.4 s
old master rejoined as replica2.0 s

Upgrade notes:

  • Upgrade replicas before the master. A master on the new protocol answers older replicas with 426 Upgrade Required; they stay readable but stale until upgraded. Each replica re-downloads once after the upgrade.
  • workflow_events gains a seq column, added automatically on start.
  • Behavior changes, all listed below: replicas refuse more writes, the SQL console refuses transaction control, and transient write failures now return 503 instead of 403.
  • No SDK changes.
  • Writes through the write queue are captured on the database’s pinned connection and appended to DATA_DIR/replication-log/<db>.clog (CRC-checked, fsynced) before their batch commits. A failed commit removes its entry; crash recovery drops uncommitted tails.
  • Entries form a hash chain. A replica resumes with its last position plus the chain hash at that position. A position that is pruned, ahead of the master, or on a different history (a failover onto a node that missed writes) gets a full download instead of a mixed history.
  • Replicas mirror the master’s log entry for entry, so after a promotion the remaining replicas catch up incrementally from the new master. A master restart costs replicas nothing.
  • Entries are kept for REPLICATION_LOG_RETENTION_HOURS (24); a replica offline longer re-downloads once. A database no replica has asked for is not logged, so standalone nodes pay nothing.
  • Full downloads use SQLite’s backup API: snapshots include writes still in the WAL, and replicas restore in place, so open handles stay valid. Previously a replica’s first full download could leave services on a closed database.
  • Writes from the SQL console and from migration scripts now go through the write queue, so they replicate. Inside them, transaction control (BEGIN, COMMIT, ROLLBACK, savepoints), ATTACH/DETACH and pragma assignments are refused: each request already runs in its own transaction.
  • A changeset carrying columns or tables a replica lacks is no longer silently skipped: the replica syncs the schema first.
  • Workflow events carry a per-run seq assigned when they commit, so a promoted replica hands workers the same event order as the old master.
  • Hot promote/demote: POST /api/admin/promote and POST /api/admin/demote (superuser token) switch a node’s role in-process: write path, syncer, changeset journal, pipeline dispatcher, cron, workflow gateway, webhooks, graph index.
  • Workflow execution is master-only: workflow gateway, scheduler, cron dispatch, webhooks and realtime run on the master alone. Replicas answer workflow writes and queue polls with 503.
  • workflows.db replicates by default as a passive copy, so a promoted replica starts with every run, queued job, timer and definition. Jobs the dead master had handed out are reclaimed within 30–60 s and redelivered. REPLICA_INCLUDE_WORKFLOWS=false keeps it node-local.
  • Replicas no longer write locally. Every write to a replicated database is refused below the HTTP layer, so a replica never holds rows its master never saw. Logins still work on replicas. Registration, OAuth account linking, admin init and API key creation now return 403 there and must go to the master; before, they silently created diverging rows.
  • Exactly-once replay: a write carrying X-Tango-Idempotency-Key records the key in the same transaction; a repeat answers 200 {"duplicate":true} without writing. Keys replicate with the data and are pruned after IDEMPOTENCY_KEY_RETENTION (2h), which must exceed tango’s journal retention.
  • Older replicas (pre-log protocol) get 426 Upgrade Required, logged once per replica every few minutes.
  • Security: /api/admin/replicas* now requires an admin. Previously any authenticated user could mint replica tokens.
  • Replicas report their positions to the master after every sync (POST /api/replica/sync-complete); every node exposes its own at GET /api/replica/local-status.
  • Transient failures are retryable: a write hitting a node that is shutting down, a full write queue, a replica, or a busy database now returns 503 with Retry-After instead of 403, which clients treated as final.
  • Write-queue batches take the write lock up front (BEGIN IMMEDIATE). A batch that read before writing could fail with “database is locked” when another connection committed in between; SQLite reports that without waiting.
  • Deleting a database released neither its replication session nor its connection handle, which could hang shutdown or crash the server. Sessions are now released before their connection closes.
  • The Docker images are built with SQLite sessions enabled. Without them a replica’s first snapshot failed, so containers could not replicate.
  • The Docker images are published for linux/amd64 and linux/arm64.

A new component (tango/): one endpoint in front of a vski master and its replicas.

  • Routing: writes go to the master. Reads go to the least-busy healthy node, except the workflow and automation plane (workflows, pipelines, cron, webhooks, plugins, secrets) and the realtime and workflow websockets, which stay on the master. Sessions (/sso, logins, /api/auth, /api/admins, /api/keys) and admin scripts (SQL console, migrations) go to the master and are never journaled.
  • Write journal: every write is appended to a Redis Stream before it is forwarded, with an idempotency key the master stores with the write. On failover tango replays, in order, every entry the new master may lack, including writes the old master acknowledged but never replicated. Writes whose client saw a failure are dropped, never replayed.
  • Failover: tango fences the old master, hot-promotes the replica with the most entries applied, then replays. In strict mode (default) writes get 503 + Retry-After during the window; in absorb mode they are journaled and answered 202.
  • Read freshness: a replica that falls behind the master by more than TANGO_MAX_READ_LAG (30s) leaves the read pool until it catches up. Optional read-your-writes (TANGO_READ_YOUR_WRITES) pins a client’s reads to the master briefly after its write.
  • Actuators carry out role changes:
    • docker: Engine API, role files read by containers at start; replicas sync through tango, so promotions need no reconfiguration
    • kubernetes (tango v1.0.2 or later): a role ConfigMap with one entry per pod and lease election between tango replicas; replicas sync through tango’s management Service, as with docker. The vski-cluster Helm chart installs the stack
    • local: child processes, for development
    • manual
  • Management API (:8001): status, manual promotion, replay.

Measured (tango/bench, three nodes on one host, k6 open model), p95 added by tango:

500/s1,000/s2,000/s4,000/s
reads (point, list)+0.2 ms+0.2–0.3 ms+0.1–0.2 ms+0.1 ms
record creates+0.1 ms+0.1 ms+5.7 ms+8.0 ms

No errors at any rate. On writes, most of the cost is the idempotency key vski stores with each write; the benchmark breaks the cost down.

  • New end-to-end suites cover multi-replica convergence, master restarts, hot promotion and switchover, incremental catch-up after promotion, workflow state across failover, and divergence handling. vski’s full end-to-end suite (47 tests) passes through tango, and tango’s failover test proves acknowledged-but-unreplicated writes survive exactly once.
  • Replication.md documents the changeset log, upgrade order, replica read-only rules and the new settings; tango ships its own README and benchmark guide.