Connecting an LLM to your documents and your APIs is not building enterprise intelligence.
It is the first step — and the most insufficient one.
Introduction
An industrial services company deployed, in 2025, what its management considered a procurement agent: an LLM connected to the ERP, the CRM, the contracts repository, and the purchasing inbox. In the demo it worked well: it answered questions about orders, drafted purchase requests, and summarized three weeks of emails with a supplier in seconds.
Three months later, the agent was no longer used. It did not fail dramatically: it did not invent orders with made-up invoice numbers (or almost never), and no one could point to a specific error that justified switching it off. It failed in a more insidious way: it did not know which of the two “master contracts” of a supplier was the current one, it could not distinguish whether a customer in the CRM was the same as an account in the ERP, it answered with the 2024 discount policy because that was the better-indexed PDF, and when a buyer asked “check whether this order violates any restriction,” it returned a plausible summary instead of a verification.
The CIO summed it up in a phrase we have heard in many rooms: “The model is intelligent. The problem is that it doesn’t understand our company.”
This article is the first in a series of twenty in which we will break that phrase down. The thesis we will defend over the coming days is a single one:
The main challenge of enterprise AI is not having increasingly intelligent models. It is providing them with the infrastructure necessary to understand an organization.
The problem
An LLM is an extraordinary language processor. Given a text, it predicts what reasonably could follow. With enough of your documentation in the context window, it will seem to understand your business. But what the model does is not comprehension: it is statistical interpolation over what you have shown it.
The difference matters because a company is not a corpus of documents. A company is:
- a set of data spread across ERP, CRM, databases, APIs, documents, emails, tickets, and legacy systems that sometimes have no API;
- a set of meanings: each field, each code, each term has an internally agreed interpretation that no PDF documents completely;
- a set of relations: who owns what, what depends on what, which contract covers which product, who is responsible for which asset;
- a set of events: negotiations, decisions, incidents, supplier changes, corrections that only exist in people’s memory or in email threads;
- a set of rules and constraints: policies, permissions, regulatory requirements, risk thresholds;
- and a set of actions that those decisions must trigger in real systems, with traceability and accountability.
None of those things lives in the documents you gave the model. It lives in the systems, in the relations between systems, in time, and in decisions. Connecting an LLM to a document database gives it access to a two-dimensional fraction of the company: what was written, not what the company is.
And here is the point that is worth underlining: the problem is not the model. The models will keep improving; that is a fact, not a promise. But no model, however large, will reliably infer that “Customer 49281” in the CRM and “ACME LTD” in the ERP are the same entity, or that master contract B replaced A on March 1, or that the buyer looking at an order does not have permission to approve discounts above 15%. That is not general intelligence: it is situated knowledge of a concrete organization. And situated knowledge has to be built.
The concept
We have used metaphors for years for this missing layer: “semantic layer,” “knowledge layer,” “context layer.” They are useful as marketing labels, but not as architecture: they describe one piece (the semantics, the context) as if it were the whole.
In this series we will use a term that is broader and, at the same time, more constrained:
Enterprise Cognitive Infrastructure (ECI): the layer of interconnected systems —semantics, knowledge, memory, context, reasoning, action, and governance—that enables AI (models and agents) to understand, remember, and operate on the knowledge of an organization with traceability, control, and accountability.
Three notes on this working definition, which we will refine across the series and which we should not treat as final:
- It is an architecture, not a product. The ECI is not bought as a box. It is built from components with names (knowledge graph, integration pipelines, typed memory, context engine, business rules, data governance) that exist today, partly, separately. The work is designing how they integrate.
- It is a layer, not a destination. It sits between the enterprise systems (the data source) and the intelligent applications (agents and assistants). It does not replace the ERP or the data warehouse: it puts them at service in a new way.
- It has measurable properties. Comprehension (does it know what each piece of data represents?), connection (does it know who is who and how they relate?), retrieval (does it bring the right information at the right time?), reasoning (does it correctly divide what the model reasons about from what the infrastructure resolves?), and auditable action (can you explain why it decided what it decided?). Each of these has known modes of failure, which we will go through.
The reason for naming this layer is practical: when the problem has a name, the budget, the team, and the architectural design also get a name. “Let’s connect the LLM to the documents” does not survive the first architecture meeting. “Let’s build the knowledge and context layer on which the agents will operate” does.
Architecture
In its most schematic form, the ECI is organized in layers. The reader will build this diagram across the series; today we only need to see its silhouette:
APPLICATIONS / USERS
↓
AGENTS
↓
REASONING
↓
┌────────────┴────────────┐
↓ ↓
CONTEXT MEMORY
↓ ↓
└──────────┬──────────────┘
↓
KNOWLEDGE (graph + RAG + semantic search)
↓
┌──────────┴──────────┐
↓ ↓
ONTOLOGY ENTITY RESOLUTION
↓ ↓
└──────────┬──────────┘
↓
SEMANTICS
↓
DATA
↓
ENTERPRISE SYSTEMS
Running transversally across the whole stack —permissions, provenance, audit, human oversight— goes governance. And in the agent layer, the action: tools, workflows, and APIs that turn decisions into operations.
Today only one thing matters: the model (the LLM) is not in this diagram as a layer. It is inside the agent, as the reasoning engine. The rest of the diagram is what the model needs so it doesn’t have to guess.
Use case
Let’s return to the procurement agent. Let’s break down its failure with the order we will use across the whole series: starting with the business problem, not the technology.
- Problem. Buyers lose time locating scattered information (contracts, current prices, supplier history) and inconsistent approvals generate off-policy discounts.
- Decision the agent should support: validate a proposed order against the current master contract and the purchasing policies, and propose the approval.
- Necessary knowledge: what a master contract is and which one is current; what a purchasing restriction is; who can approve which amount; which supplier is the one appearing in the order.
- Data: ERP records (orders, suppliers, accounts), contracts in PDF, negotiation emails, discount policy, product catalogs.
- Relations: order → references → contract; contract → covers → products; product → depends on → components; supplier → supplies → component. The “Customer #49281” in the CRM and “ACME LTD” in the ERP → same entity.
- Context: who is asking (and what permissions they have), what date it is (which contract is current now), at what stage of the process the order is, what level of risk the decision carries.
- Memory: the previous conversations with that supplier, the last contract review, the failed-delivery incident from last quarter, the decision the director made in a similar case.
- Reasoning: apply the constraint “maximum 15% discount outside promo” to the specific order; check that the price is within the contractual range; cross the order date with the contract’s validity.
- Action: generate the approval request in the ERP with the attached justification; escalate to the buyer if the discount exceeds the threshold.
- Infrastructure: everything above —entity identity, relation graph, memory, context, rules, action tools, audit— is what the agent needed. The model only provided the last piece.
The lesson is not “use a knowledge graph.” The lesson is that the failure was not in the language layer: it was that none of the layers 5 to 10 existed for the agent.
Trade-offs
Building this layer has costs that are worth acknowledging from day one:
- Cost. A knowledge layer well done costs more than connecting a RAG to a vector store. It is paid for with architecture, pipelines, and governance.
- Complexity. Each component (graph, memory, context, rules) is a subsystem with its own operation. It is not a three-week project.
- Latency. Retrieving, contextualizing, and verifying adds steps to the path of the answer. You have to design what is resolved hot and what is precomputed.
- Maintenance. A company’s knowledge changes: contracts, suppliers, org charts, policies. What is not maintained breaks down (we will look at this in detail later in the series).
- Security. Centralizing knowledge is centralizing the attack surface. Permissions have to be applied at the knowledge layer, not only at the application layer.
- Scalability. Not every question needs the same effort. The architecture must be selective: bringing the whole graph for a “how much does this part cost?” query would be absurd.
- Honest limitation: the ECI does not turn a broken process into a good one. If the organization does not know which master contract is current because no one has ever validated it, the infrastructure will make that uncertainty visible —which is a lot— but it will not resolve it on its own.
Implication for the business
For the CIO or the CTO, the mindset shift is concrete: the AI budget should not start at the model or the agent, but at the question “what layers of comprehension exist in this organization today that AI can use?” For the CDO, the ECI is a data project with a new consumer: the agent demands data with identity, semantics, and provenance at a level traditional reporting never demanded. For operations, the promise is not “less work” in the short term, but consistent decisions: the same criterion, applied the same way, at three in the afternoon and at eleven at night.
And for the AI team, the message is the most important one in the series: if your agent fails in production and you don’t know why, the problem probably is not in the prompt.
Conclusion
We have named the layer that was missing in the procurement agent: an Enterprise Cognitive Infrastructure, composed of semantics, knowledge, memory, context, reasoning, action, and governance. What remains to be seen is how it is built, piece by piece. And for that it makes sense to start from the ground up: from the data, which is where everything begins and where almost everything falls apart.
Next article
If the procurement agent needed to understand the company, the first ingredient was the humblest: its data. But there is a classic distinction that almost every AI project confuses, and confusing it explains half of the failures we have described: the difference between having data and having knowledge. Tomorrow: From data to knowledge: the real challenge of enterprise AI.
Sources
- Russell, S. and Norvig, P. Artificial Intelligence: A Modern Approach — the classic distinction between knowledge representation and language processing.
- Working definition of the ECI: proposed by Cirtra (2026), developed in this article and in the series. It is not an industry standard; it is presented as an architectural framework.
