Retrieving a document answers “what is written?”. An agent that
decides needs more: to know what has happened, what has been decided, and why.
Retrieval is the first step, not the knowledge.
Introduction
Yesterday we closed the knowledge graph as the structure for relational
knowledge. And we mentioned, in passing, its complement: the RAG, the mechanism
that retrieves knowledge from documents. Today it is its turn — and with
an important correction that has crept into all enterprise AI
conversations: RAG is not memory, and it is not knowledge. It is retrieval.
And confusing the two explains why so many “RAG-powered” systems fail in
production: they solved retrieval well and did not realize that
retrieval was the smallest of their needs.
The problem
An insurance company reviews claims with an AI assistant. The assistant
has an impeccable RAG: the entire corpus of policies, general terms and
procedures is indexed, retrieval is fast and the citations are
correct. In the demo, it answers well to “what does the fire policy cover?”.
In production, it falls apart on the real decisions:
- “Should we pay this claim?” — The assistant retrieves the
general term that says theft is covered “under certain
conditions”. It does not know that this specific policy has an endorsement that
excludes partial theft, nor that there is a previous claim of the same type
pending. - “Is this loss-adjustment report coherent?” — It retrieves the
adjustment procedure, but it does not know that the adjuster who signed it is in a
sanction process for a previous report. - “What did we decide in the similar case of 2024?” — It has no way
of knowing. The RAG retrieves documents, not past decisions.
The pattern is the same: the RAG brings the text, but the decision needs the
text + the concrete state + the history + the rules applied. And
those last four things are not documents: they are state, memory and
reasoning.
The concept
RAG (Retrieval-Augmented Generation) is a retrieval mechanism:
before generating an answer, the system retrieves relevant information
from an external source (documents, databases, graphs) and injects it into
the model’s context to ground the answer.
Let us break down the chain, because that is where the weak points are:
- Indexing. Turn the corpus into something retrievable (chunks +
embeddings, or structure). - Retrieval. Find what is relevant. Variants:
– Semantic search (embedding similarity): good for “similar texts”.
– Keyword/BM25: good for exact terms.
– Hybrid search: combines both; it is the de facto standard today.
– Graph retrieval: retrieve by traversing the knowledge graph
(what we saw yesterday).
– Contextual retrieval: enrich each chunk with context before
indexing, so the similarity is more precise. - Reranking. Reorder the retrieved results with a
relevance model before injecting them. - Injection into context. Put the fragments into the model’s window.
- Generation. The model answers over that context.
What RAG does not do:
- It does not know what has happened: it retrieves what is written, not the current state.
- It does not remember: it does not retain decisions, conversations or experiences
over time (that is memory, tomorrow). - It does not contextualize the decision: it does not know who is asking, what permissions
they have, or at what point of the process the task is (that is context). - It does not reason over rules: it does not verify that the answer complies with the
domain’s constraints (that is reasoning, day 14). - It does not act: it executes nothing; it only produces text.
The central question this article sets for the rest of the series:
Are we retrieving documents, or are we retrieving the knowledge necessary
to make a decision?
For the former, RAG is enough. For the latter, RAG is a piece — an
important piece, but a piece — of a larger infrastructure.
Architecture
In the series diagram, the RAG is the retrieval mechanism alongside
knowledge:
KNOWLEDGE
↑
┌───────────────────────────────┐
│ KNOWLEDGE GRAPH + RAG │ ← today: retrieval
│ (structured) (prose) │ from both worlds
└───────────────────────────────┘
↑
ONTOLOGY + ENTITY RESOLUTION
↑
SEMANTICS / DATA
RAG is not a layer apart: it is the process by which knowledge
(prose and structure) reaches the agent at the moment of the decision. And that
process is what we increasingly call today, in a broader sense, retrieval: not “searching in a vector store”, but “getting, with the right
source, what the decision needs”.
Use case
The insurance company, with its claims review. We break it down:
- Problem. The adjuster needs a grounded recommendation on
a complex claim (partial theft with an exclusionary endorsement and a
previous claim), and today it takes days to gather policy, endorsements, history and
prior criteria. - Decision. Recommend paying, rejecting or escalating the claim, with the
justification. - Knowledge needed. What the base policy covers; what the
endorsement excludes; whether there is a previous claim of the same type; what criteria were applied in
similar cases; what amount threshold forces escalation. - Data. Policy, endorsements, general terms (PDF), the
loss-adjustment report, the policyholder’s claims history. - Relationships. Policy → has → endorsement; endorsement → excludes → coverage;
policyholder → has → previous claims; claim → type → historical
criteria. - Context. The claim is high-amount (above the
escalation threshold); the policyholder is a strategic customer; there is a
fraud investigation open on the type of theft. - Memory. In 2024 a similar claim was settled by paying half
due to the depreciation clause; the decision and its justification are in
the case history. - Reasoning. Apply: the base coverage covers theft, BUT the
endorsement excludes partial theft → the claim falls into the exclusion;
cross with the previous claim (possible pattern); cross with the historical
criteria (depreciation). - Action. Recommend “reject the partial-theft coverage by
endorsement; pay the non-excluded part with depreciation according to the
historical criteria; escalate by amount” with the chain of justification. - Infrastructure. RAG over policy+endorsements+terms (prose) +
relationship graph policy→endorsement→coverage (structured) + memory of
the 2024 case + amount and investigation context + reasoning that
applies the rules. The RAG brought the text; the other layers brought the
decision.
The case shows the rule of the series: the technology (RAG, graph, memory)
appears as a consequence of the need (to recommend a grounded
decision), not as a starting point.
Trade-offs
- Cost. A decent RAG (hybrid + reranking + contextual) costs more
than a naïve vector store. And each retrieval improvement is a recurring
operational cost. - Complexity. The retrieval→rerank→inject chain has many parameters
(chunk size, number of results, relevance threshold) that have to
be tuned per domain. There is no universal configuration. - Latency. Hybrid retrieval + reranking adds steps to the
answer. For real-time decisions you have to precompute or
cache. - Quality of the source. The RAG retrieves what is there; if the corpus has
obsolete versions or contradictions, it retrieves those too. The
disambiguation of versions is a governance problem (day 18), not a
retrieval problem. - Security. Retrieving sensitive documents requires that permissions be
applied in the retrieval, not just in the application. An RAG that does
not filter by permissions is a data leak with an assistant’s face. - Honest limitation: the RAG does not solve the contradiction
problem. If two documents say different things, retrieving both
does not solve which one is true: that requires governance (freshness, source
authority) and, sometimes, human intervention.
Implication for the company
For the CDO, the RAG is the gateway to knowledge in prose, but
it is not the knowledge: it is the mechanism that brings it. And as a mechanism, it has
an owner (the AI team) and metrics (retrieval precision,
relevance) that have to be measured.
For the AI team, the lesson is one of discipline: do not confuse “the RAG
works” (it retrieves well) with “the agent decides well” (it applies well). They are
different metrics, and the second is the one that matters for the business.
For top management, the message is one of expectations: an assistant “with
RAG” that answers well to questions about documents is not, yet, a
system that makes decisions. The distance between the two is the rest of this
series.
Conclusion
RAG is retrieval, not memory and not knowledge: it brings the text that the
decision needs, but the decision also needs the state, the history,
the context and the rules. Today we dismantled the retrieval chain
(indexing, retrieval, reranking, injection, generation) and we set the
question that guides the rest of the series: do we retrieve documents, or
do we retrieve the knowledge necessary to decide?
And the answer points to a next step we have been mentioning: the
agent does not only need what is written, it needs what has
happened. And “what has happened” is not a document: it is memory.
Next article
Tomorrow: The Memory of an Enterprise Agent: What an AI Should Remember.
We will see the memory typologies (working, episodic, semantic,
procedural and organizational) and why an agent that does not remember is an
agent that starts from zero in every conversation.
Sources
- Lewis, P. et al. (2020). “Retrieval-Augmented Generation for
Knowledge-Intensive NLP Tasks”. — Origin of the RAG concept. - Anthropic (2024). “Introducing Contextual Retrieval”. — Contextual and
hybrid retrieval; reduces failed retrievals by up to 67 %
(figure reported by Anthropic in its documentation). - Microsoft Research (2024). GraphRAG. — Retrieval over knowledge
graphs.
