Skip to content

Tango

Tango is the VSKI cluster proxy and the entrypoint of a production deployment. Clients (SDKs, the web dashboard, workers) talk to tango and only to tango; tango talks to the vski master and its read replicas.

flowchart TD
C["Clients<br/>(SDKs, dashboard, workers)"] --> T["tango<br/>proxy + failover"]
T -->|"writes (journaled to Redis first)"| M["vski master<br/>(read/write)"]
T -->|"reads, least-connections"| M
T -->|"reads, least-connections"| R1["vski replica"]
T -->|"reads, least-connections"| R2["vski replica"]
M -->|"changeset sync (pulled through tango :8001)"| R1
M -->|"changeset sync (pulled through tango :8001)"| R2
T <-.-> J[("Redis<br/>write journal")]

Everything the proxy does is described from the operator’s point of view in Replication; the short version:

  • Routing: writes always reach the master; reads spread least-connections over the master and every healthy, fresh replica. Realtime websockets and the automation plane (workflows, pipelines, cron, webhooks, plugins, secrets) are pinned to the master, so through tango the whole platform (realtime included) behaves exactly like a single node.
  • Read freshness: a replica that trails the master by more than TANGO_MAX_READ_LAG (default 30s) leaves the read pool until it catches up. Stale replicas never answer reads.
  • Read-your-writes: set TANGO_READ_YOUR_WRITES (the chart defaults to 10s) and a client’s reads go to the master for that window after its write.
  • Write journal: every write is appended to a Redis Stream before it is forwarded, stamped with an idempotency key the master records in the same transaction. After a failover, journal entries replay exactly once.
  • Failover: when the master dies, tango fences it and hot-promotes the most-current replica: vski flips the node’s role in-process, in milliseconds, without a restart. Workflow state replicates ahead of time, so a promoted master resumes runs, queued jobs, timers and definitions. With two tango replicas (the chart default), a Kubernetes lease decides which one may change roles.
  • Replica sync: replicas point MASTER_URL at tango’s management port, which forwards replication to whichever node is master, so a promotion needs no reconfiguration.
  • Management API on :8001: GET /tango/status (cluster state), POST /tango/promote (manual failover) and the replica sync forwarding. Keep this port on an internal network.

In a clustered deployment, store uploaded files in S3 (or any S3-compatible store: AWS S3, MinIO, Garage, R2) instead of node-local disks. Every node reads and writes the same bucket, so:

  • replicas never sync file payloads from the master; the local file-sync journal turns itself off,
  • any node can serve /files/... downloads at full speed,
  • file backup, versioning and lifecycle become the bucket’s job instead of yours.
VariableMeaning
S3_ENDPOINThost[:port]; empty means AWS S3
S3_REGIONregion (AWS)
S3_BUCKETbucket name; setting this is what switches storage to S3
S3_ACCESS_KEY / S3_SECRET_KEYcredentials
S3_FORCE_PATH_STYLEtrue for MinIO and most self-hosted S3

Local storage keeps working (replicas then sync files over HTTP as part of replication; see File Sync), but in production S3 is the recommended mode.

The vski-cluster chart installs the full stack: a vski StatefulSet (vski-0 starts as master, the rest are replicas; tango records every role change in a ConfigMap), tango (two replicas with lease-election RBAC), the client Service that points at tango, tango’s internal management Service (the replicas’ MASTER_URL), Redis for the journal, and secrets.

Terminal window
git clone https://vski.sh/x/k8s.git
cd k8s
helm upgrade --install vski ./vski-cluster \
--namespace vski --create-namespace \
--set s3.bucket=my-vski-files \
--set s3.endpoint=s3.eu-central-1.amazonaws.com \
--set s3.region=eu-central-1 \
--set s3.accessKey=AKIA... \
--set s3.secretKey=...

The images are published for linux/amd64 and linux/arm64: vski.sh/x/vski (vX.Y.Z is the API server, vX.Y.Z-standalone adds the web UI; the chart defaults to its appVersion, v0.17.7) and vski.sh/x/tango (v1.0.2; Kubernetes needs v1.0.2 or later). For the web UI, add --set image.tag=v0.17.7-standalone.

The JWT secret is generated on install and reused by every upgrade. It is also kept when the release is uninstalled: unless secretsKey is set, it encrypts stored workflow secrets too, and losing it loses them. To bring your own, create a secret with the key jwt-secret (and optionally secrets-key) and set existingSecret.

All client and worker traffic goes to the <release>-vski Service (port 3000), which selects the tango pods. The vski pods themselves should only be reached by tango; networkPolicy.enabled=true enforces that, and limits tango’s management port to the vski pods and the peers in networkPolicy.managementFrom.

ValueDefaultPurpose
replicaCount3vski instances: 1 master + N-1 read replicas
image.repository / image.tagvski.sh/x/vski / appVersionvski image
tango.image.repository / tango.image.tagvski.sh/x/tango / v1.0.2tango image
tango.replicaCount2tango replicas; a lease picks the one that may fail over
tango.bufferModestrictstrict: 503 during failover; absorb: journal and 202 (use with one tango replica)
tango.journalRetention30mjournal window; must exceed worst-case replication lag
tango.readYourWrites10spin a client’s reads to the master after its write
tango.healthInterval / tango.failureThreshold2s / 3node probes; failover starts three rounds after the master is down
tango.maxReadLag30sreplicas further behind leave the read pool
service.sessionAffinityClientIPkeeps a client on one tango replica, where its read-your-writes pin lives
syncInterval5replica poll interval in seconds
replication.logRetentionHours24changeset-log window on each PVC
replication.includeWorkflowstruereplicate workflow state so a promoted master resumes it
replication.idempotencyKeyRetention2hmust stay above tango.journalRetention
storage.className / storage.size"" / 10GiPVC per vski pod
s3.*emptyS3 file storage (see above); empty keeps local files
existingSecret / jwtSecret / secretsKeyemptycluster secrets; empty generates them once
redis.enabledtruein-chart Valkey with AOF on a PVC; for production set enabled=false and point redis.external.address at managed Redis
networkPolicy.enabledfalseonly tango may reach vski pods; management port only from vski pods
Terminal window
helm upgrade vski ./vski-cluster --reuse-values --set replicaCount=5

The StatefulSet adds pods at the highest ordinals. Each new pod reads its role from the role map (no entry: anything but ordinal 0 is a replica), downloads the databases through tango, and tango adds it to the read pool once it is fresh. No other change: clients keep the same address.

Lower replicaCount the same way; the highest-ordinal pods go first and their PVCs stay (a re-added pod reuses its volume and resyncs incrementally, or re-downloads once if it was gone longer than replication.logRetentionHours).

One care point: if the current master is one of the pods you are removing (it stopped being vski-0 after a past promotion), promote another node first. Promotions go to the tango replica that holds the lease; the other one answers 409 with the leader’s name:

Terminal window
kubectl -n vski port-forward svc/vski-tango-mgmt 8001:8001 &
curl -s localhost:8001/tango/status | jq -r .leader_id # the lease holder
kubectl -n vski port-forward pod/<leader_id> 8002:8001 &
curl -X POST localhost:8002/tango/promote \
-H 'Content-Type: application/json' \
-d '{"target": "http://vski-vski-0.vski-vski-headless:3000"}'

Then scale down.

The chart runs two tango replicas: both proxy traffic, and a lease decides which may change roles. For more proxy throughput raise tango.replicaCount and put any ingress or load balancer in front of the <release>-vski Service. The journal lives in Redis; read-your-writes pins live in each tango replica, which is why the client Service keeps a client on one replica (service.sessionAffinity=ClientIP). Behind an ingress that sends everything from one ingress pod to one tango replica; set service.sessionAffinity=None to spread the load instead.

The cluster survives the master’s pod dying: tango promotes the most-current replica automatically (about ten seconds from the loss to a new master with the default probe settings; the promotion itself takes milliseconds), and the replicas follow it through tango’s management Service. If the master’s node is unreachable, its pod cannot finish terminating: tango waits up to 15 seconds for it, then promotes anyway, since it routes nothing to that node any more and the role map already says replica. vski pods prefer distinct nodes and start independently, so one dead node never blocks the others. What you should plan for is the volume: durable PVCs are what make a master restart lossless, so prefer a StorageClass with real replication over hostPath-style provisioners.

Upgrade replicas before the master: a master on the changeset-log protocol answers an older replica with 426 Upgrade Required, and the replica stays readable but stale (and out of tango’s read pool) until upgraded. The StatefulSet’s rolling update goes highest-ordinal-first, so while vski-0 is the master a chart upgrade applies replicas before it touches the master:

Terminal window
helm upgrade vski ./vski-cluster --reuse-values \
--set image.tag=v0.17.7 \
--set tango.image.tag=v1.0.2

The rolling update restarts the master too. If that restart outlasts failure detection (about ten seconds), tango fails over to a replica and the restarted pod rejoins as a replica; nothing is lost either way.

Keep replication.idempotencyKeyRetention (default 2h) above tango.journalRetention.

Terminal window
# cluster state: roles, health, per-db positions, journal, lease leader,
# split-brain flag
kubectl -n vski port-forward svc/vski-tango-mgmt 8001:8001
curl localhost:8001/tango/status | jq
# the role map tango maintains (pod -> master | replica)
kubectl -n vski get configmap vski-vski-roles -o yaml

Manual failover goes to the lease holder (see Remove read capacity); without a target, tango picks the most-current replica.

Watch two things: replication lag per replica (/tango/status) against TANGO_MAX_READ_LAG, and journal trim age against tango.journalRetention. A replica consistently near the lag bound is your signal to check its sync interval, its storage, or to rebalance reads. If split_brain is ever true, two nodes report master: tango holds writes until only one does.

The highest-ordinal replica doubles as the backup node: stop it, snapshot its volume, start it again; production never pauses. Check first that it is still a replica: after a promotion it can be the master. See Backup for the procedure (cron example with docker, CronJob sketch for Kubernetes) and for restoring.

  • Replication: the sync protocol, roles, headers, failover mechanics in detail
  • Home servers: the same chart on Docker Desktop or Rancher, nothing to configure
  • Docker deployments: a full cluster with tango on a single docker host, no Kubernetes
  • Backup: snapshot a stopped replica, restore it anywhere
  • tango repository: source, configuration reference, e2e and failover tests