Semantic Search for Git Repositories

Note See source code at https://vski.sh/x/embeddings_tutorial

A tutorial on building a semantic code search service using VSKI’s vector embeddings and Ollama. This guide walks you through creating a CLI tool that indexes your codebase and enables natural language search across it.

What You’ll Build

A CLI tool with two commands:

• index <path> - Indexes a directory, tracks git changes, generates embeddings
• search <query> - Performs semantic search, returns ranked results with line numbers

Architecture

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   CLI Tool      │────▶│   VSKI Server   │────▶│  SQLite + Vec   │
│   (Python)      │     │   (Docker)      │     │  (Embedded)     │
└─────────────────┘     └─────────────────┘     └─────────────────┘
          │
          │ HTTP
          ▼
┌─────────────────┐
│     Ollama      │
│  (Embeddings)   │
│   (Local)       │
└─────────────────┘

Prerequisites

• Docker
• Python 3.11+
• Git
• Ollama (running locally)

Quick Start

1. Start Ollama and Pull Model

# Ensure Ollama is running
ollama serve

# Pull the embedding model
ollama pull nomic-embed-text

2. Start VSKI Server

docker compose up -d

This starts VSKI server on http://localhost:8347

Create admin account (first time only)

Navigate to installer http://localhost:8347/installer and create your admin.

Or use curl:

curl -X POST http://localhost:8347/api/admins/init \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@example.com","password":"yourpassword"}'

3. Initialize the Database

# Install the VSKI command line tool (includes the Python SDK)
pip install "vski-sdk[cli]" --extra-index-url https://vski.sh/api/packages/x/pypi/simple/

# Login via the command line
vski login --url http://localhost:8347 --email admin@example.com --password yourpassword

# Create database
vski db create semantic_search

# Apply the schema (the vski/ YAML from Step 2 below)
vski apply

4. Index Your Code

# Index current directory
python src/main.py index .

# Index a specific directory
python src/main.py index ./src

5. Search

python src/main.py search "error handling"
python src/main.py search "database connection"
python src/main.py search "authentication middleware"

How It Works

Indexing Process

1. File Discovery - Walks the directory tree, detects text files via MIME type
2. Git Integration - Gets current branch, commit hash, and detects uncommitted changes
3. Chunking - Splits files into ~500 token segments with 50 token overlap
4. Line Tracking - Records line numbers for each chunk
5. Embedding Generation - Sends chunks to Ollama for vector embedding
6. Storage - Stores chunks and embeddings in VSKI

Search Process

1. Query Embedding - Converts search query to vector via Ollama
2. Vector Search - Queries VSKI for similar embeddings
3. Ranking - Returns results sorted by similarity score
4. Formatting - Displays file paths, line ranges, and content previews

Project Structure

docs/semantic_search/
├── README.md              # This file
├── docker-compose.yml     # Service definitions
├── vski/                  # Declarative schema (IaC YAML)
│   └── semantic_search/
│       └── collections.yaml
└── src/
    ├── main.py            # CLI entry point
    ├── indexer.py         # Indexing logic
    ├── embedder.py        # Ollama integration
    ├── searcher.py        # Search logic
    └── utils.py           # Utility functions

Step-by-Step Tutorial

Step 1: Docker Compose Setup

The docker-compose.yml defines the VSKI service:

services:
  vski:
    image: vski.sh/x/vski:latest-standalone
    ports:
      - "8347:8347"
    environment:
      - SERVER_PORT=8347
      - DATA_DIR=/app/data
      - JWT_SECRET=dev-secret-change-in-production
    volumes:
      - vski_data:/app/data

volumes:
  vski_data:

Ollama runs locally (not in Docker) for better GPU access.

Step 2: Define the Schema

The code_chunks collection stores indexed code segments. It is declared in the IaC YAML (see IaC) and created by vski apply via a server-side migration:

vski/semantic_search/collections.yaml

version: 1
database: semantic_search

collections:
  code_chunks:
    type: base
    fields:
      - {name: file_path, type: text, required: true}
      - {name: content, type: text, required: true}
      - {name: line_start, type: number, required: true}
      - {name: line_end, type: number, required: true}
      - {name: branch, type: text}
      - {name: commit_hash, type: text}
      - {name: modified, type: bool}

Step 3: Embedding Generation

The embedder module handles communication with Ollama (its /api/embeddings endpoint takes {"model": ..., "prompt": text} and returns {"embedding": [...]}):

src/embedder.py

import httpx

OLLAMA_URL = "http://localhost:11434"
EMBEDDING_MODEL = "nomic-embed-text"


async def generate_embedding(text: str) -> list[float]:
    async with httpx.AsyncClient() as http:
        response = await http.post(
            f"{OLLAMA_URL}/api/embeddings",
            json={"model": EMBEDDING_MODEL, "prompt": text},
        )
        response.raise_for_status()
        return response.json()["embedding"]

Step 4: File Indexing

The indexer walks directories, chunks text, and stores embeddings:

src/indexer.py

from vski import VskiClient

from embedder import generate_embedding
from utils import chunk_text, discover_text_files, get_git_info


async def index_directory(client: VskiClient, dir_path: str) -> None:
    files = discover_text_files(dir_path)   # list of pathlib.Path
    git_info = get_git_info(dir_path)

    for file in files:
        content = file.read_text(encoding="utf-8", errors="ignore")

        for chunk in chunk_text(content, str(file)):
            embedding = await generate_embedding(chunk["content"])

            record = await client.collection("code_chunks").create({
                "file_path": chunk["file_path"],
                "content": chunk["content"],
                "line_start": chunk["line_start"],
                "line_end": chunk["line_end"],
                "branch": git_info["branch"],
                "commit_hash": git_info["commit"],
                "modified": git_info["has_changes"],
            })

            await client.embeddings.upsert("code_chunks", record["id"], embedding)

Step 5: Semantic Search

The searcher queries embeddings and formats results:

src/searcher.py

from vski import VskiClient

from embedder import generate_embedding


async def search_code(
    client: VskiClient,
    query: str,
    limit: int = 10,
) -> list[dict]:
    embedding = await generate_embedding(query)

    results = await client.embeddings.search(
        "code_chunks", embedding, limit=limit, threshold=0.7
    )

    return [
        {
            "file_path": r["record"]["file_path"],
            "line_start": r["record"]["line_start"],
            "line_end": r["record"]["line_end"],
            "content": r["record"]["content"],
            "distance": r["distance"],
        }
        for r in results["results"]
    ]

Step 6: Command Line Entry Point

The main module ties everything together with a plain argparse CLI:

src/main.py

import argparse
import asyncio
import os

from vski import VskiClient

from indexer import index_directory
from searcher import search_code


async def main(args: argparse.Namespace) -> None:
    async with VskiClient(os.getenv("VSKI_URL", "http://localhost:8347")) as client:
        res = await client.admins.auth_with_password(
            "admin@example.com", "yourpassword"
        )
        client.set_token(res["token"])
        client.set_db("semantic_search")

        if args.command == "index":
            await index_directory(client, args.path)
        else:  # search
            results = await search_code(client, args.query)
            display_results(results)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(prog="semantic-search")
    sub = parser.add_subparsers(dest="command", required=True)

    index = sub.add_parser("index")
    index.add_argument("path")

    search = sub.add_parser("search")
    search.add_argument("query")

    asyncio.run(main(parser.parse_args()))

Configuration

Environment Variables

Variable	Default	Description
VSKI_URL	http://localhost:8347	VSKI server URL
VSKI_TOKEN	-	Admin JWT token
OLLAMA_URL	http://localhost:11434	Ollama server URL
EMBEDDING_MODEL	nomic-embed-text	Ollama model for embeddings

Chunking Parameters

Parameter	Default	Description
CHUNK_SIZE	500	Target tokens per chunk
CHUNK_OVERLAP	50	Token overlap between chunks

Advanced Usage

Index Only Changed Files

The indexer automatically detects uncommitted changes via git. You can filter:

# Only index modified files
files = discover_text_files(dir_path)
modified_files = [f for f in files if is_modified(str(f))]

Re-indexing

To re-index from scratch:

# Clear existing chunks
page = await client.collection("code_chunks").get_list(1, 10000)
for item in page["items"]:
    await client.collection("code_chunks").delete(item["id"])

# Re-index
await index_directory(client, dir_path)

Multiple Branches

Index multiple branches for comparison:

git checkout feature-branch
python src/main.py index .

git checkout main
python src/main.py index .

Then search with branch filter:

results = await client.collection("code_chunks").get_list(
    1, 20, filter='branch = "feature-branch"'
)

Troubleshooting

Ollama Model Not Found

# Pull the model manually
docker-compose exec ollama ollama pull nomic-embed-text

VSKI Authentication Failed

# Re-login
vski login --url http://localhost:8347

Slow Indexing

The indexer processes files sequentially. For large codebases, consider:

• Reducing chunk size
• Indexing specific directories
• Using a GPU-enabled Ollama setup

Next Steps

• Add file watching for automatic re-indexing
• Implement incremental updates (only index changed files)
• Add support for more embedding models
• Build a web UI for search
• Add syntax highlighting in previews

Resources

• VSKI Documentation
• Vector Embeddings Guide
• Ollama Documentation
• nomic-embed-text Model