From Vectors to a Graph

01 · In one paragraphThe migration

The Digital Twin's retrieval layer is being rebuilt. The old system splits knowledge base documents into ~900-character chunks, embeds each one, and searches them by vector similarity. The new system stores full sections as the retrieval unit, promotes Projects, Skills, Methods, Technologies, and Concepts to first-class nodes with explicit edges between them, and ranks results with a hybrid score that combines vector similarity with graph signals. The payoff is queries that couldn't be answered before — "which projects use knowledge graphs?" — and retrieval that returns complete thoughts instead of fragments. The cost is one additional database to operate, one new query language to write, and a manual entity-curation step until the canonicalization pipeline is fully self-preserving.

02 · Why migrateThree problems with the current system

Each problem in the table below is a real failure mode observed in production, not a hypothetical. The Neo4j move is what the new architecture does to address it.

03 · Three movesThe architecture, mapped to the problems

Every other choice in the migration plan exists to support one of these three moves. When you get lost, come back here.

Move 1 — Make the section a first-class retrieval unit

Each H2 block in a KB document becomes a Section node carrying its complete text and its own embedding. Section boundaries match the existing ChromaDB ingestion logic exactly, which keeps the before/after evaluation honest.

Move 2 — Promote entities to typed nodes with typed edges

Skills, Methods, Technologies, Publications, and Concepts become nodes. Projects connect to them through named relationships: DEMONSTRATES, USES_METHOD, USES_TECHNOLOGY, DOCUMENTED_IN. Sections connect to entities through MENTIONS edges that carry a count and a contextual snippet.

Move 3 — Replace L2 distance with a multi-signal hybrid score

Vector similarity stays in the mix at 60% weight, but the final ranking also rewards sections that explicitly describe a Project (+25%), sections rich in entity mentions (+10%), and longer/more substantial sections (+5%). The full formula lives in section 6.

04 · The schemaWhat lives in the graph

Two tiers. The document hierarchy (Document → Section → Chunk) is where text lives. The entity network (Project at center, connecting Skill / Method / Technology / Publication) is what the text is about. The two tiers are bridged by MENTIONS edges (entities cited within sections) and DESCRIBED_IN edges (projects pointing back to the sections that describe them).

Node types — quick reference

Note on properties: role (Skill) and stage (Method) are edge properties, not node properties. They live on the DEMONSTRATES and USES_METHOD relationships respectively, so the same Skill can play a "core" role on one Project and a "supporting" role on another.

Relationship types — quick reference

By the numbers — what's actually in the graph

Real counts from canonical_entities.json as of the most recent canonicalization run. These numbers are the actual order of magnitude you're operating at — useful both for tuning decisions and for talking about the system honestly.

Canonicalization in action

The three-phase canonicalization pipeline (Decision 2) compresses 279 raw extracted entity names down to 167 canonical nodes — a 1.92× reduction. The compression happens because the same idea appears across multiple project walkthroughs under slightly different names, and the LLM batch phase collapses those into a single canonical with alt_labels.

A few real merges the pipeline produced:

Skill roles · 57 canonical skills

The role property — carried on the DEMONSTRATES edge — distributes as:

Method stages · 42 canonical methods

The stage property — carried on the USES_METHOD edge — distributes as:

The ingestion-heavy distribution is honest about where the methodological work goes in these projects — most of the design effort is in getting data into a usable shape before any retrieval or generation happens.

MENTIONS edges per Section · across 121 sections with mention data

How many entities (Projects, Skills, Concepts combined) each Section mentions:

Median is 9 mentions per section, mean is 9.8. About 75% of sections sit at 6+ mentions — well above the +0.10 bonus cap of 5. The signal is effectively binary in production right now: either a section has <5 mentions and gets a partial bonus, or it has 5+ and gets the full bonus. The cap likely needs to move to 10–15 to actually differentiate sections. Flagged in section 12.

Per-project entity density · raw counts from project_entities

Before canonicalization, this is the entity footprint each project contributed. Higher numbers don't mean better projects — they mean more methodological surface area to articulate.

Sensitivity tier · across 121 sections with mention data

Browse the canonical entities

Live data from canonical_entities.json — all 167 canonical entity nodes. Search across names and alt_labels, or filter by type. Click any row to see full alt_labels and (for concepts) the description.

05 · Same query, two pathsHow a question travels through each system

The clearest way to internalize the difference is to follow a single query through both pipelines. Here are two contrasting cases — one where both systems work but the experience differs, and one where only the graph approach can answer the question at all.

Case A — "Tell me about the Resume Graph Explorer"

A factual question with a clear target. Both systems can answer it; the difference is in what comes back.

CALL db.index.vector.queryNodes('section_embeddings', 10, $query_embedding)
YIELD node AS section, score AS vector_score
WHERE section.sensitivity IN $allowed_tiers

// Reward sections that DESCRIBE the project, not just mention it
OPTIONAL MATCH (section)<-[:DESCRIBED_IN]-(p:Project)
OPTIONAL MATCH (section)-[:MENTIONS]->(entity)

WITH section, vector_score,
     count(DISTINCT p) AS projects_described,
     count(DISTINCT entity) AS entities_mentioned

WITH section,
     (vector_score * 0.6 +
      CASE WHEN projects_described > 0 THEN 0.25 ELSE 0 END +
      toFloat(CASE WHEN entities_mentioned > 5 THEN 5 ELSE entities_mentioned END) / 5 * 0.10 +
      (CASE WHEN section.char_count > 2000 THEN 0.05 ELSE 0 END)) AS final_score
ORDER BY final_score DESC LIMIT 5
RETURN section.full_text, section.name, final_score

Case B — "Which projects use knowledge graphs?"

This is the kind of query that vector search structurally cannot answer well. The ChromaDB approach returns whatever section happens to mention knowledge graphs most prominently — useful, but not a list. The graph approach traverses the relationship and returns the actual answer.

# Embed "Which projects use knowledge graphs?"
# Return 10 chunks ordered by similarity to that embedding
# LLM reads chunks and tries to extract a list of projects
# Failure mode: misses projects that don't say "knowledge graph" verbatim
results = collection.query(query_embeddings=[emb], n_results=10)

MATCH (skill:Skill {name: "Knowledge Graphs"})<-[:DEMONSTRATES]-(project:Project)
RETURN project.title, project.summary
ORDER BY project.title

06 · Hybrid scoringWhy these four weights, and what they're tuning

Section ranking in Neo4j is a weighted sum of four signals, each normalized to [0, 1] so that count-based components can't dominate the embedding similarity. Maximum possible score ≈ 1.0.

What each weight is doing

• 0.60 · vector similarity — the workhorse. If the query is semantically close to the section's content, this carries most of the signal.
• 0.25 · project-described boost — distinguishes sections that describe a project from sections that merely mention one. This is the move that solves the "specificity" problem in ranking.
• 0.10 · entity-richness signal — capped at 5 entities. Rewards sections that sit at intersections in the graph (e.g., a section discussing both a Skill and a Project is more useful than one that mentions only a name in passing).
• 0.05 · length tiebreaker — small bonus for substantial sections. Mostly a tiebreaker to avoid surfacing thin sections when richer alternatives exist.

07 · Architectural decisionsFour resolved choices and why they matter

These decisions are marked do not re-open without new evidence in the migration plan. Each one shapes what the rest of the work has to do.

08 · Simple queriesFor understanding what's in the graph

These are the queries to run in the Neo4j browser when you want to feel out the shape of what's loaded. None of them are evaluation queries — they're for inspection and intuition.

What sections exist, grouped by document

MATCH (d:Document)-[:HAS_SECTION]->(s:Section)
RETURN d.title AS document, collect(s.name) AS sections
ORDER BY d.title

Every project, with the sections that describe it

MATCH (p:Project)-[:DESCRIBED_IN]->(s:Section)
RETURN p.title, collect(s.name) AS describing_sections
ORDER BY p.title

Health check — projects with no DESCRIBED_IN edge

// If this returns rows, those projects are invisible to the +0.25 ranking boost
MATCH (p:Project)
WHERE NOT (p)-[:DESCRIBED_IN]->()
RETURN p.id, p.title

MENTIONS edge distribution — sanity check

// If most sections have 0-1 mentions, entity extraction is too sparse.
// If most have 20+, the +0.10 cap will saturate everywhere.
MATCH (s:Section)-[m:MENTIONS]->()
RETURN s.name, count(m) AS mention_count
ORDER BY mention_count DESC LIMIT 20

Entity inventory

MATCH (n)
WHERE n:Skill OR n:Method OR n:Technology OR n:Concept
RETURN labels(n)[0] AS type, count(*) AS total
ORDER BY total DESC

09 · Demo queriesFor showing the difference between before and after

These are the queries to use when explaining the migration to someone who hasn't seen it before — in an interview, in a LinkedIn post, in a walkthrough. Each one is chosen because the graph approach does something the vector approach structurally can't.

The Cypher behind the demo queries

MATCH (skill:Skill {name: "Knowledge Graphs"})<-[:DEMONSTRATES]-(p:Project)
RETURN p.title, p.summary

MATCH (p1:Project {id: "resume-graph-explorer"})-[:USES_METHOD]->(m:Method)
      <-[:USES_METHOD]-(p2:Project)
WHERE p1 <> p2
RETURN p2.title, collect(m.name) AS shared_methods
ORDER BY size(shared_methods) DESC

MATCH (s:Section)-[m:MENTIONS]->(c:Concept {name: "Bayesian reasoning"})
RETURN s.name, m.count, m.context
ORDER BY m.count DESC

10 · Metrics definedWhat the evaluation harness measures

The migration plan calls for five test batteries comparing ChromaDB baseline against Neo4j. Each metric below maps to one of them.

11 · GlossaryThe jargon, in one place

12 · Adjust laterPlaces to revisit once the system is running

None of these is blocking. They're flagged here so they don't disappear into folklore.