Tango
Tango is the VSKI cluster proxy and the entrypoint of a production deployment. Clients (SDKs, the web dashboard, workers) talk to tango and only to tango; tango talks to the vski master and its read replicas.
flowchart TD C["Clients<br/>(SDKs, dashboard, workers)"] --> T["tango<br/>proxy + failover"] T -->|"writes (journaled to Redis first)"| M["vski master<br/>(read/write)"] T -->|"reads, least-connections"| M T -->|"reads, least-connections"| R1["vski replica"] T -->|"reads, least-connections"| R2["vski replica"] M -->|"changeset sync (pulled through tango :8001)"| R1 M -->|"changeset sync (pulled through tango :8001)"| R2 T <-.-> J[("Redis<br/>write journal")]Everything the proxy does is described from the operator’s point of view in Replication; the short version:
- Routing: writes always reach the master; reads spread least-connections over the master and every healthy, fresh replica. Realtime websockets and the automation plane (workflows, pipelines, cron, webhooks, plugins, secrets) are pinned to the master, so through tango the whole platform (realtime included) behaves exactly like a single node.
- Read freshness: a replica that trails the master by more than
TANGO_MAX_READ_LAG(default 30s) leaves the read pool until it catches up. Stale replicas never answer reads. - Read-your-writes: set
TANGO_READ_YOUR_WRITES(the chart defaults to10s) and a client’s reads go to the master for that window after its write. - Write journal: every write is appended to a Redis Stream before it is forwarded, stamped with an idempotency key the master records in the same transaction. After a failover, journal entries replay exactly once.
- Failover: when the master dies, tango fences it and hot-promotes the most-current replica: vski flips the node’s role in-process, in milliseconds, without a restart. Workflow state replicates ahead of time, so a promoted master resumes runs, queued jobs, timers and definitions. With two tango replicas (the chart default), a Kubernetes lease decides which one may change roles.
- Replica sync: replicas point
MASTER_URLat tango’s management port, which forwards replication to whichever node is master, so a promotion needs no reconfiguration. - Management API on
:8001:GET /tango/status(cluster state),POST /tango/promote(manual failover) and the replica sync forwarding. Keep this port on an internal network.
File storage: use S3
Section titled “File storage: use S3”In a clustered deployment, store uploaded files in S3 (or any S3-compatible store: AWS S3, MinIO, Garage, R2) instead of node-local disks. Every node reads and writes the same bucket, so:
- replicas never sync file payloads from the master; the local file-sync journal turns itself off,
- any node can serve
/files/...downloads at full speed, - file backup, versioning and lifecycle become the bucket’s job instead of yours.
| Variable | Meaning |
|---|---|
S3_ENDPOINT | host[:port]; empty means AWS S3 |
S3_REGION | region (AWS) |
S3_BUCKET | bucket name; setting this is what switches storage to S3 |
S3_ACCESS_KEY / S3_SECRET_KEY | credentials |
S3_FORCE_PATH_STYLE | true for MinIO and most self-hosted S3 |
Local storage keeps working (replicas then sync files over HTTP as part of replication; see File Sync), but in production S3 is the recommended mode.
Deploy with Helm
Section titled “Deploy with Helm”The vski-cluster chart
installs the full stack: a vski StatefulSet (vski-0 starts as master, the
rest are replicas; tango records every role change in a ConfigMap), tango
(two replicas with lease-election RBAC), the client Service that points at
tango, tango’s internal management Service (the replicas’ MASTER_URL),
Redis for the journal, and secrets.
git clone https://vski.sh/x/k8s.gitcd k8s
helm upgrade --install vski ./vski-cluster \ --namespace vski --create-namespace \ --set s3.bucket=my-vski-files \ --set s3.endpoint=s3.eu-central-1.amazonaws.com \ --set s3.region=eu-central-1 \ --set s3.accessKey=AKIA... \ --set s3.secretKey=...The images are published for linux/amd64 and linux/arm64:
vski.sh/x/vski (vX.Y.Z is the API server, vX.Y.Z-standalone adds the
web UI; the chart defaults to its appVersion, v0.17.7) and
vski.sh/x/tango (v1.0.2; Kubernetes needs v1.0.2 or later). For the web
UI, add --set image.tag=v0.17.7-standalone.
The JWT secret is generated on install and reused by every upgrade. It is
also kept when the release is uninstalled: unless secretsKey is set, it
encrypts stored workflow secrets too, and losing it loses them. To bring
your own, create a secret with the key jwt-secret (and optionally
secrets-key) and set existingSecret.
All client and worker traffic goes to the <release>-vski Service (port
3000), which selects the tango pods. The vski pods themselves should only be
reached by tango; networkPolicy.enabled=true enforces that, and limits
tango’s management port to the vski pods and the peers in
networkPolicy.managementFrom.
Values worth knowing
Section titled “Values worth knowing”| Value | Default | Purpose |
|---|---|---|
replicaCount | 3 | vski instances: 1 master + N-1 read replicas |
image.repository / image.tag | vski.sh/x/vski / appVersion | vski image |
tango.image.repository / tango.image.tag | vski.sh/x/tango / v1.0.2 | tango image |
tango.replicaCount | 2 | tango replicas; a lease picks the one that may fail over |
tango.bufferMode | strict | strict: 503 during failover; absorb: journal and 202 (use with one tango replica) |
tango.journalRetention | 30m | journal window; must exceed worst-case replication lag |
tango.readYourWrites | 10s | pin a client’s reads to the master after its write |
tango.healthInterval / tango.failureThreshold | 2s / 3 | node probes; failover starts three rounds after the master is down |
tango.maxReadLag | 30s | replicas further behind leave the read pool |
service.sessionAffinity | ClientIP | keeps a client on one tango replica, where its read-your-writes pin lives |
syncInterval | 5 | replica poll interval in seconds |
replication.logRetentionHours | 24 | changeset-log window on each PVC |
replication.includeWorkflows | true | replicate workflow state so a promoted master resumes it |
replication.idempotencyKeyRetention | 2h | must stay above tango.journalRetention |
storage.className / storage.size | "" / 10Gi | PVC per vski pod |
s3.* | empty | S3 file storage (see above); empty keeps local files |
existingSecret / jwtSecret / secretsKey | empty | cluster secrets; empty generates them once |
redis.enabled | true | in-chart Valkey with AOF on a PVC; for production set enabled=false and point redis.external.address at managed Redis |
networkPolicy.enabled | false | only tango may reach vski pods; management port only from vski pods |
Scaling
Section titled “Scaling”Add read capacity
Section titled “Add read capacity”helm upgrade vski ./vski-cluster --reuse-values --set replicaCount=5The StatefulSet adds pods at the highest ordinals. Each new pod reads its role from the role map (no entry: anything but ordinal 0 is a replica), downloads the databases through tango, and tango adds it to the read pool once it is fresh. No other change: clients keep the same address.
Remove read capacity
Section titled “Remove read capacity”Lower replicaCount the same way; the highest-ordinal pods go first and
their PVCs stay (a re-added pod reuses its volume and resyncs incrementally,
or re-downloads once if it was gone longer than
replication.logRetentionHours).
One care point: if the current master is one of the pods you are removing
(it stopped being vski-0 after a past promotion), promote another node
first. Promotions go to the tango replica that holds the lease; the other
one answers 409 with the leader’s name:
kubectl -n vski port-forward svc/vski-tango-mgmt 8001:8001 &curl -s localhost:8001/tango/status | jq -r .leader_id # the lease holder
kubectl -n vski port-forward pod/<leader_id> 8002:8001 &curl -X POST localhost:8002/tango/promote \ -H 'Content-Type: application/json' \ -d '{"target": "http://vski-vski-0.vski-vski-headless:3000"}'Then scale down.
Scale tango
Section titled “Scale tango”The chart runs two tango replicas: both proxy traffic, and a lease decides
which may change roles. For more proxy throughput raise
tango.replicaCount and put any ingress or load balancer in front of the
<release>-vski Service. The journal lives in Redis; read-your-writes pins
live in each tango replica, which is why the client Service keeps a client
on one replica (service.sessionAffinity=ClientIP). Behind an ingress that
sends everything from one ingress pod to one tango replica; set
service.sessionAffinity=None to spread the load instead.
Surviving a host failure
Section titled “Surviving a host failure”The cluster survives the master’s pod dying: tango promotes the most-current replica automatically (about ten seconds from the loss to a new master with the default probe settings; the promotion itself takes milliseconds), and the replicas follow it through tango’s management Service. If the master’s node is unreachable, its pod cannot finish terminating: tango waits up to 15 seconds for it, then promotes anyway, since it routes nothing to that node any more and the role map already says replica. vski pods prefer distinct nodes and start independently, so one dead node never blocks the others. What you should plan for is the volume: durable PVCs are what make a master restart lossless, so prefer a StorageClass with real replication over hostPath-style provisioners.
Upgrades
Section titled “Upgrades”Upgrade replicas before the master: a master on the changeset-log protocol
answers an older replica with 426 Upgrade Required, and the replica stays
readable but stale (and out of tango’s read pool) until upgraded. The
StatefulSet’s rolling update goes highest-ordinal-first, so while vski-0
is the master a chart upgrade applies replicas before it touches the master:
helm upgrade vski ./vski-cluster --reuse-values \ --set image.tag=v0.17.7 \ --set tango.image.tag=v1.0.2The rolling update restarts the master too. If that restart outlasts failure detection (about ten seconds), tango fails over to a replica and the restarted pod rejoins as a replica; nothing is lost either way.
Keep replication.idempotencyKeyRetention (default 2h) above
tango.journalRetention.
Operations
Section titled “Operations”# cluster state: roles, health, per-db positions, journal, lease leader,# split-brain flagkubectl -n vski port-forward svc/vski-tango-mgmt 8001:8001curl localhost:8001/tango/status | jq
# the role map tango maintains (pod -> master | replica)kubectl -n vski get configmap vski-vski-roles -o yamlManual failover goes to the lease holder (see Remove read capacity); without a target, tango picks the most-current replica.
Watch two things: replication lag per replica (/tango/status) against
TANGO_MAX_READ_LAG, and journal trim age against
tango.journalRetention. A replica consistently near the lag bound is your
signal to check its sync interval, its storage, or to rebalance reads. If
split_brain is ever true, two nodes report master: tango holds writes
until only one does.
Backups
Section titled “Backups”The highest-ordinal replica doubles as the backup node: stop it, snapshot its volume, start it again; production never pauses. Check first that it is still a replica: after a promotion it can be the master. See Backup for the procedure (cron example with docker, CronJob sketch for Kubernetes) and for restoring.
Where to go next
Section titled “Where to go next”- Replication: the sync protocol, roles, headers, failover mechanics in detail
- Home servers: the same chart on Docker Desktop or Rancher, nothing to configure
- Docker deployments: a full cluster with tango on a single docker host, no Kubernetes
- Backup: snapshot a stopped replica, restore it anywhere
- tango repository: source, configuration reference, e2e and failover tests