VSKI v0.9.0

Embeddings: model tags, vector fetch, and a fixed vec0 layout

The embeddings API grows the primitives downstream ML consumers need: provenance (which embedding model produced these vectors) and export (the actual vectors, not just distances).

Model tags

Upsert accepts an optional model form field. The first tagged write pins it for the collection in the new _vec_metadata system table (tenant db, next to the <collection>_vec table - deliberately not collection options, which dashboard PATCHes can rewrite). Later upserts declaring a different model are rejected with 409; search responses and the new info endpoint echo the pinned tag, so a consumer can refuse embeddings from a model it wasn’t built for:

curl -F recordId=$RID -F model=llama-backbone-q8 -F vector=@v.f32 \
  localhost:3000/api/collections/notes/embeddings

New endpoints

• GET /api/collections/:name/embeddings - {collection, model, dimensions, metric, count}; a collection with no vectors yet reports nulls and count: 0.
• GET /api/collections/:name/embeddings/:recordId - the exact stored vector as a JSON float array (float32 values, lossless round-trip).
• POST /api/collections/:name/embeddings/fetch - {"ids": [...]} → {"vectors": [{recordId, vector}], "missing": [...]}, indexed point lookups (max 1000 ids).

Search additionally validates the query dimension against the pinned one (clear 400 instead of a deep vec0 error) and can be combined with vec_to_json/hex through the admin SQL console as before.

The vec0 layout fix (migration required)

The old virtual-table layout (rowid INTEGER PRIMARY KEY with record_id as an unindexed metadata column) had two problems: point lookups by record_id were full scans, and INSERT OR REPLACE never conflicted - every re-upsert of the same record appended a duplicate row that then duplicated search hits.

The canonical layout is now record_id TEXT PRIMARY KEY, which indexes record_id in the shadow rowids table (true point queries) and makes re-upserts replace. vec0 tables cannot be ALTERed or renamed, so BootstrapDB rebuilds legacy tables transactionally on first boot after the upgrade - copying vectors through a temp table, deduplicating with last-write-wins - and seeds _vec_metadata (model: null, dimensions parsed from the old DDL, metric: "l2").

Migration notes

• The rebuild copies every vector once; on multi-GB collections expect a one-time boot proportional to table size. It runs inside a single transaction per table and is idempotent.
• Duplicate rows accumulated by old re-upserts are collapsed to the latest vector per record_id.
• Distance semantics are unchanged and now documented: sqlite-vec’s default euclidean L2 over unnormalized float32 vectors, ascending (the vendored sqlite-vec l2 kernel returns the sqrt - plain distance, not the square).
• make test-vec runs the vec0-dependent Go tests (plain go test compiles without the sqlite_vec extension).

Compatibility

API is additive (one new optional upsert field, three new endpoints, two new response fields). The TS client and the Python SDK gain matching methods. Existing clients see one behavior change: re-upserting a record no longer duplicates it in search results - that was the bug.

The vendored sqlite driver fork also gained an unconditional #cgo LDFLAGS: -lm in sqlite3.go: cgo compiles every .c file in the package directory, so sqlite-vec.c (and its sqrt calls) links in every build, not just sqlite_vec-tagged ones - untagged go test binaries failed to link on toolchains where libm is not implicit.