CodeLens: AI-Powered Codebase Q&A & Hybrid Search

CodeLens is a semantic code search, retrieval, and question-answering system. It allows developers to index entire git repositories and ask natural language questions about the codebase structure, classes, and logic.

---

🚀 Key Features & Architecture

CodeLens uses a state-of-the-art hybrid RAG (Retrieval-Augmented Generation) pipeline:

                  ┌──────────────────────┐
                  │   Git Repository     │
                  └──────────┬───────────┘
                             │ (Clone)
                             ▼
                  ┌──────────────────────┐
                  │ Tree-Sitter Parser   │
                  └──────────┬───────────┘
                             │ (Extract Code Blocks)
                             ▼
                  ┌──────────────────────┐
                  │ Presidio Sanitizer   │
                  └──────────┬───────────┘
                             │ (PII Redacted)
                             ▼
        ┌────────────────────┴────────────────────┐
        ▼                                         ▼
┌──────────────┐                          ┌──────────────┐
│ Gemini Embed │                          │  Sparse FTS  │
└───────┬──────┘                          └──────┬───────┘
        │ (Dense Embedding)                      │ (Text Tokens)
        ▼                                        ▼
┌────────────────────────────────────────────────────────┐
│                   Supabase Database                    │
└────────────────────────┬───────────────────────────────┘
                         │
                         ▼
┌────────────────────────────────────────────────────────┐
│           Hybrid Search Retrieval (RRF Fusion)         │
└────────────────────────┬───────────────────────────────┘
                         │ (Top Candidates)
                         ▼
┌────────────────────────────────────────────────────────┐
│             Cohere Rerank (v4.0-pro)                   │
└────────────────────────┬───────────────────────────────┘
                         │ (Top Ranked Contexts)
                         ▼
┌────────────────────────────────────────────────────────┐
│             Gemini generation (Flash-Lite)             │
└────────────────────────────────────────────────────────┘

1. Ingestion & AST Parsing: Codebase repositories are cloned, scanned, and parsed using Tree-sitter (supporting .py, .js, and .ts) to extract logical code segments (functions/methods).
2. Compliance Sanitizer:
   • PII Protection: Uses Presidio Analyzer & Anonymizer to detect and redact sensitive entities (emails, keys, phones) before indexing or generation.
   • Prompt Injection Defense: Filters inputs against known prompt injection and jailbreak patterns.
3. Observability: Traces query performance, token usage, and latencies via MLflow and Langfuse.
4. Hybrid Retrieval Pipeline:
   • Dense vector search: Queries Supabase vector store (pgvector) using gemini-embedding-2.
   • Sparse text search: Triggers native Postgres Full-Text Search (FTS) matching tokens across content, function names, and summaries.
   • Fusion & Rerank: Combines sparse and dense search lists via Reciprocal Rank Fusion (RRF), then reranks candidates using Cohere rerank-v4.0-pro.
5. Generation: Builds a context-aware system prompt and generates structured answers using Gemini 3.1 Flash-Lite.

---

🛠️ Getting Started

1. Installation

Clone the repository and set up a virtual environment:

python -m venv venv
source venv/Scripts/activate  # On Windows
pip install -r requirements.txt
pip install -r requirements-dev.txt

2. Configure Environment Variables

Create a .env file in the root directory (based on .env.example):

# Gemini API Key
GOOGLE_API_KEY=your-api-key
GEMINI_MODEL=gemini-3.1-flash-lite
GEMINI_EMBED_MODEL=gemini-embedding-2

# Supabase Credentials
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_KEY=your-anon-key
DATABASE_URL=postgresql://postgres:password@db.your-project.supabase.co:5432/postgres

# Cohere API Key
COHERE_API_KEY=your-cohere-key

3. Run the FastAPI Application

Start the uvicorn development server:

python -m uvicorn src.api.main:app --host 127.0.0.1 --port 8000

• Access the Web UI: http://127.0.0.1:8000/
• Interactive API Docs: http://127.0.0.1:8000/docs

4. Run Automated Tests

python -m pytest -v tests/

5. Run RAG Quality Evaluations (Ragas)

Execute Ragas metrics (Faithfulness, Relevancy, Precision, Recall) and log results to MLflow:

python scripts/run_eval.py

---

🔮 Future Enhancements & Technical Roadmap

To further build out CodeLens, we are following a structured phase-by-phase implementation plan:

Phase 1: Robustness, Optimization, & Quality (Completed)

• Database-Backed Hybrid Search: Moved sparse search from in-memory python indexing (rank-bm25) to native Postgres Full-Text Search (FTS) to eliminate memory bottlenecks on larger codebases.
• Unit & Integration Tests: Established comprehensive pytest suites covering parser, sanitization, compliance redaction, hybrid search, and FastAPI client routing.
• RAG Evaluation Suite: Implemented test datasets, Gemini wrappers, and Ragas metrics computation logged directly to MLflow.

Phase 2: Ingestion & Parsing Expansion (Completed)

• AST Node Expansion: Parse entire classes, global constants, package imports, docstrings, and config files (JSON/YAML/TOML) in parser.py rather than just function definitions.
• Multi-Language Support: Enable Tree-sitter and structural fallback configurations for Go, Rust, Java, C++, HTML, and CSS.
• HTML & CSS Landing Page Understanding: Implement fallback text-structure parsing for .html and .css files. This allows CodeLens to answer layout, markup, and styling questions.
• AI Ingestion Summaries: Enrich code chunk metadata and embeddings with AI logic summaries during ingestion.

Phase 3: Code Semantics & Call-Graph Navigation (Completed)

• Symbol Call-Graph Indexing: Trace how functions interact. Extracts symbol calls (e.g. store.New, jwt.GenerateToken) and parent class scopes during AST parsing.
• Context-Aware Retrieval: Pass parent scope and symbol call references into LLM prompt contexts for accurate call hierarchy explanations.

Phase 4: Advanced Security & Guardrails (Completed)

• Secret Scanning: Automated credential scanning (secret_scanner.py) during ingestion to redact AWS keys, API tokens, RSA/SSH private keys, JWTs, and database credentials.
• Lightweight LLM Guardrails: Multi-layer prompt injection and jailbreak classifiers (sanitizer.py) blocking prompt leakage and instruction overrides.

Phase 5: Production Readiness & Ops (Completed)

• Docker Containers: Production multi-container environment configured in Dockerfile and docker-compose.yml (FastAPI + PostgreSQL pgvector).
• WebSocket Real-Time Logging: Added /api/v1/ws/ingest/logs WebSocket endpoint with real-time log streaming in static/index.html.

Phase 6: User Management, Subscriptions, & Billing (Completed)

• Authentication & Authorization: Password hashing (PBKDF2/SHA256 with salt) and signed JWT token issuance/verification (auth.py) across /auth/register, /auth/login, and /auth/me.
• Payment Gateway & Tiers: Subscription tier management (Free & Pro) and Stripe payment checkout session generator (billing.py) at /billing/checkout.