Embedded, Local-First Hybrid GraphRAG Retrieval Engine & MCP Server
[:EXTENDS], [:CONTRADICTS], [:PREREQUISITE_FOR]"]
TypedEdges --> BaseGraph
style T1 fill:#0f172a,stroke:#3b82f6,color:#e2e8f0
style T2 fill:#0f172a,stroke:#8b5cf6,color:#e2e8f0
style T3 fill:#7c2d12,stroke:#f97316,color:#e2e8f0
linkStyle default stroke:#38bdf8,stroke-width:2px;
```
---
## ๐ Key Architectural Principles
- **Zero-Daemon, Embedded Storage:** Runs entirely in-process using embedded C++ and Apache Arrow engines: **LadybugDB** for property graph traversals, **LanceDB** for hybrid vector/BM25 search, and **SQLite WAL** for sub-0.5ms L1 query caching. No Docker, no network databases.
- **Dialect-Agnostic Core:** Storage, retrieval, and MCP tools operate on abstract knowledge primitives (`Note`, `Tag`, `Folder`, `LINKS_TO`). Syntax parsing is completely encapsulated in pluggable dialects.
- **Model Context Protocol (MCP):** Serves as a high-precision retrieval data plane for AI agents (**Claude Code**, **Cursor**, **OpenCode**) without exposing raw filesystem mutation risks.
- **Scientific Quality Gating:** Retrieval accuracy is benchmarked with zero-cost IR metrics (MRR, Context Recall@5) guarding against semantic regressions.
---
## ๐ ๏ธ Quickstart
### 1. Installation
Install PKMRAG into an isolated global environment using `pipx` or `uv tool`:
```bash
# Recommended for end users
pipx install pkmrag
# or
uv tool install pkmrag
```
For local development or cloning from source:
```bash
git clone https://github.com/dev-hashemi/pkmrag.git
cd pkmrag
uv sync # dev environment
uv tool install --editable . # also expose the `pkmrag` CLI globally, from this checkout
```
`--editable` links the global CLI to your working tree, so code changes take effect on the next restart โ no reinstall. (Drop it for a fixed snapshot, then re-run with `--force` to update.)
### 2. Verify Storage & Engine Health
```bash
pkmrag doctor
```
```text
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ PKMRAG v0.9.0 โ System Health & Diagnostics โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
In-Process Data Plane Verification
โโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโ
โ Component โ Status โ Version โ Latency โ Verification Details โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ LadybugDB Property โ PASS โ 0.20.4 โ 0.3ms โ Cypher transactional โ
โ Graph โ โ โ โ read/write verified โ
โ LanceDB Vector Engine โ PASS โ 0.38.0 โ 6.3ms โ Arrow-backed vector โ
โ โ โ โ โ index verified โ
โ Ollama Local Engine โ PASS โ active โ 1.2ms โ Local daemon detected โ
โโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโ
All systems operational. Engines ready for PKMRAG.
```
### 3. Ingest and Search in 30 Seconds
```bash
pkmrag ingest /path/to/vault
pkmrag search "how does cache invalidation work?" --near "Cache Architecture.md"
```
---
## ๐ Core Capabilities
### Ingestion & Incremental Synchronization
Ingest notes, links, tags, and semantic vectors with SHA-256 delta sync:
```bash
pkmrag ingest /path/to/vault # Ingest graph topology + semantic vectors
pkmrag ingest /path/to/vault --target vector # Target specific plane (all, graph, or vector)
pkmrag ingest /path/to/docs --dialect commonmark # Explicitly select markdown dialect
pkmrag ingest /path/to/vault --rebuild # Force a clean rebuild
```
### Hybrid & Graph-Boosted Retrieval
Combines dense semantic similarity (`BAAI/bge-small-en-v1.5`), Tantivy BM25 keyword matching, Reciprocal Rank Fusion ($k=60$), and graph proximity multipliers:
```bash
pkmrag search "distributed consensus" # Hybrid search (dense + BM25 via RRF)
pkmrag search "consensus" --near "Paxos Algorithm.md" # Boosted by graph proximity to anchor note
pkmrag search "raft" --mode dense --limit 5 # Explicit retrieval mode (hybrid, dense, sparse)
pkmrag search "storage" --json # Machine-readable JSON output
```
### ๐ง Semantic Gap Discovery & Inferred Relationships
Detects unlinked note pairs with high semantic similarity ($\ge 0.80$) but graph distance $\ge 3$, classifying missing edges via local or remote LLMs:
```bash
pkmrag discover /path/to/vault --dry-run # Preview semantic gaps without calling LLM
pkmrag discover /path/to/vault --provider ollama # 100% offline local inference (Llama 3.2, Qwen 2.5)
pkmrag discover /path/to/vault --threshold 0.85 # Custom cosine similarity threshold
```
Inferred edges are stored separately in the LadybugDB `[:INFERRED_REL]` table with confidence, model name, and rationaleโpreserving human-curated wikilinks.
---
## ๐ Model Context Protocol (MCP) & HTTP Server
PKMRAG exposes a rich tool plane over **stdio** (for CLI/IDE agents) and **HTTP/SSE** (for Obsidian and network clients):
```bash
# Start stdio MCP server (Claude Code, Cursor, OpenCode)
pkmrag serve /path/to/vault
# Start HTTP/SSE daemon on port 3747 (dual MCP SSE + REST API plane)
pkmrag serve /path/to/vault --transport http --port 3747
# One-command registration for Claude Code:
claude mcp add pkmrag -- uv run pkmrag serve /path/to/vault
```
### Exposed Tools
- `query_vault`: Hybrid semantic + BM25 keyword retrieval with graph boost and folder/tag filters.
- `read_note`: Read note content enriched with graph context (tags, forward links, backlinks).
- `list_notes`: Browse notes in the vault with optional folder filtering and pattern matching.
- `list_tags` / `search_by_tag`: Rank and query notes by tag topology.
- `get_outline`: Extract heading hierarchy and line numbers for large notes.
- `get_note_context`: Inspect incoming backlinks, outgoing citations, 2-hop clusters, and AI-inferred relationships.
- `find_bridges`: Discover the shortest link path between two notes across the vault.
- `discover_gaps`: Uncover unlinked note pairs with high vector similarity for bridging.
- `vault_overview`: Bird's-eye view of vault notes, wikilinks, tags, and central hub notes.
- `reindex_note`: Targeted sub-40ms single-note incremental reindexing after external edits.
- `sync_vault`: Incremental delta scan updating modified notes across the vault.
---
## ๐ Obsidian Desktop Plugin (`pkmrag`)
A native desktop companion plugin (`plugins/obsidian/`) connecting Obsidian to PKMRAG:
- **Interactive Setup Wizard:** If installed before the Python core engine, the sidebar automatically guides you through a 2-step setup with 1-click command copying and live verification.
- **PKMRAG Insights Sidebar:** Real-time semantic gap recommendations, contradiction warnings with LLM citations, and structural graph context.
- **Evidence-Backed Missing Links:** Each gap card shows the similarity score, the real graph distance (`N hops away` or `No path within 10 hops`), the clickable **bridge path** connecting the two notes, and the **matching passages** from both notes that justify the suggestion.
- **One-Click Related Links:** `Add to Related` appends a folder-aware `[[Target Note]]` under the source note's `## Related` section (created automatically), never writing into whichever editor happens to be focused.
- **Navigable Graph Context:** Backlinks, outgoing links, and the 2-hop cluster are listed as clickable notes instead of bare counts, with vault-wide graph counters in the footer.
- **Instant Suggestions:** Gaps are mined from a note-scoped vector sweep, so the sidebar stays responsive no matter how large the vault is.
- **Zero-Terminal Workflow & Auto-Daemon:** Transparently starts the background engine daemon on launch and shuts down cleanly on exit.
- **In-App AI Discovery & Indexing:** Trigger deep AI relationship discovery (`โก Discover`) and full vault rebuilds directly from Obsidian without opening a shell.
- **Settings-Driven LLM & Embedding Models:** Swap between Ollama, OpenAI, Groq, or local endpoints and customize models with 1-click latency verification directly inside Obsidian Settings.
- **CodeMirror 6 Inline Indicators:** Ambient visual widgets on headings (`๐ N`, `โ ๏ธ`) without document clutter; clicking reveals the sidebar panel.
- **Proximity Concept Explorer:** Context-anchored hybrid retrieval drawer (`--near