GraphRAG is worth it when your important questions depend on relationships between entities, such as who owns what, what depends on what, or what changed when. If most questions can be answered from one passage, well-tuned vector or hybrid RAG is usually enough.
This guide explains how to tell the difference, how to build a graph for retrieval, and how to test whether it actually improves answers.
What GraphRAG Means
GraphRAG adds a knowledge graph to retrieval. Entities such as people, products, contracts or systems become nodes; relationships become edges. At query time, the system walks those edges to assemble connected evidence for the model.
The term also refers to Microsoft's GraphRAG project, which builds entity graphs and community summaries from text. The original paper focused on answering broad questions across a whole corpus.
When GraphRAG Helps and When It Does Not
| Question type | Example | Better approach |
|---|---|---|
| Single fact lookup | What is our travel per diem? | Vector or hybrid RAG |
| Multi-hop | Which customers use a service owned by the team that had last week's incident? | GraphRAG |
| Corpus-wide themes | What are the main risks raised across 2,000 audit reports? | GraphRAG with community summaries |
| Current versus superseded | What is the current version of this policy? | Graph with validity dates, or strong metadata |
| Counts and absence | Which accounts have no assigned owner? | Graph or SQL query, not embeddings |
Microsoft's Azure HorizonDB guidance gives a similar split: relationships and multi-hop reasoning favor graphs; simple lookups favor vector-only retrieval.
Six Steps to Build GraphRAG
Step 1: Collect hard questions. Gather 50 to 100 real questions users struggle with today. Tag each as lookup, multi-hop, corpus-wide or aggregate.
Step 2: Define a small schema. List the entity and relationship types those questions need. Model only what improves answers; a graph of everything is costly and noisy.
Step 3: Load structured data first. Import entities that already exist in systems of record, such as CRM accounts, HR org charts or CMDB services. This is cheaper and more accurate than extraction.
Step 4: Extract from documents where needed. Use an LLM or NER pipeline to pull entities and relationships from text, with source references on every edge. Sample and review extracted edges by hand.
Step 5: Choose query patterns. Common patterns are local search around an entity, global search over community summaries, and generated graph queries, such as Cypher, for precise counts.
Step 6: Route and combine. Use vector or keyword search to find entry nodes, then traverse the graph to collect evidence. Route lookup questions to plain hybrid RAG.
Tooling Options
• Microsoft GraphRAG: open-source indexing pipeline with local and global search modes
• Neo4j GraphRAG for Python: graph building and retrievers on a Neo4j database
• Amazon Neptune GraphRAG: managed graph storage integrated with AWS services
• Graph features in existing databases: some relational and multi-model databases now offer graph queries alongside vectors
Check each tool's current documentation for supported models, languages and limits before committing.
Worked Example: IT Incident Impact Questions
Illustrative example. An IT operations team wants an assistant to answer "Which customer-facing apps were affected by the database patch on Tuesday?"
• Vector RAG result: returns the change ticket and a few incident notes, but cannot connect the database to the apps that depend on it
• Graph schema: Service, Database, Team and Change nodes, with DEPENDS_ON, OWNS and MODIFIED edges, loaded from the CMDB
• GraphRAG flow: find the change ticket, traverse to the database, then follow DEPENDS_ON edges to services and their owners
• Answer evidence: a list of apps with the edge path behind each one, which reviewers can check
The CMDB already held these relationships, so no LLM extraction was needed for the core graph.
Cost and Effort Drivers
• Extraction: LLM-based entity extraction over large corpora can cost far more than embedding
• Updates: graphs need incremental refresh when documents and systems change; see incremental indexing for RAG
• Entity resolution: merging duplicates like "Acme Inc" and "ACME Corporation" takes rules and review
• Permissions: edges can reveal facts a user should not see, so filter by access during traversal
• Operations: a graph database is another system to secure, back up and monitor
Evaluation Checklist
• Same question set run on hybrid RAG and GraphRAG, scored separately for retrieval and answer quality
• Precision of extracted edges measured on a reviewed sample
• Multi-hop questions answered with a visible evidence path
• Permission tests show restricted entities never appear in answers
• Latency and cost per query recorded for both approaches
• Freshness test confirms superseded facts are not returned
For general retrieval metrics, see how to evaluate a RAG system.
Common Mistakes to Avoid
• Starting with the graph, not the questions: the schema should follow real queries
• Trusting extracted edges: a wrong edge creates a confident but false path
• Skipping a baseline: without a hybrid RAG comparison you cannot prove the graph helps
• Ignoring time: relationships that changed need dates, or old facts will look current
Limitations
GraphRAG adds build and running costs, and published results on accuracy gains vary by domain and dataset. We make no claim of a fixed improvement. Test on your own questions before scaling.
Next Step
If your users ask questions that cross systems, owners or time, LuMay's RAG development team can help you test Graph RAG against your current retrieval and decide whether it earns its cost.
• From Local to Global: A GraphRAG Approach (arXiv 2404.16130)
