Forward deployed engineering, Part 4: Data readiness and RAG from zero

In Part 3 we designed inside Acme Consumer Brands’ estate: where the data lives, which identity provider signs people in, which network paths are allowed. With the design agreed, the work moves to the data itself.

Acme’s question is still the one from the agents series: “What did those stockouts cost us?” The volumes and order lines sit in Databricks tables, and the agents series shows how an agent queries them safely. But in the second week of the engagement the supply team told me something that is not in any table. The penalty for a late or short delivery is set per retailer, in the supply agreement. Those agreements are PDFs in SharePoint. Some are scanned. Two retailers have more than one version live. So the cost of a stockout depends on a document, and an agent that cannot read that document can only give half an answer.

This is what the Databricks tables already hold for the first four weeks of the campaign:

what the Databricks tables already hold for the stockout question: supply_service gives units and OTIF per week, service_cost gives the penalty per week; both are one row per brand, region and week, owned by supply; the per-retailer terms behind the penalty live in the agreements

Prerequisite 2: The business terms and the data behind the agents explains every column, including how the penalty is calculated.

This part is about that other half: getting documents ready and building retrieval-augmented generation (RAG) from zero, so that an agent can answer from them and show where each answer came from.

If you prefer to watch first, here is the heart of this part as a short animated explainer.

Video: what makes documents ready for an agent, and how RAG turns them into cited answers, in three minutes. The narration is a synthetic voice; the script and animation are mine.

Where does this show up in an FDE job description?

Retrieval is one of the most consistent requirements in public FDE postings. Google Cloud’s GenAI forward deployed engineer listings ask for:

“Experience building pipelines for structured and unstructured data using both vector databases and RAG-like architectures to power enterprise AI solutions.”
The ability to “address release blockers including solving integration hurdles, data readiness issues, and state-management issues”.

A Google Cloud FDE role on the enterprise context layer adds “Tune AI agents by providing the appropriate voice, tools, and specific knowledge base required for specific brands”, and lists experience with knowledge graphs and ontologies as a plus.

The job is less about calling a model and more about giving the model the right facts.

What does “data ready” actually mean?

Here is the simple definition I use with customers: data is ready when an agent can find it, read it, trust that it is current, know who to ask when it is wrong, and respect who is allowed to see it. Each of those is a check you can run in the first week.

data readiness as five checks: findable, readable, current, owned and permitted, each shown with what it meant for Acme's supply documents

Findable. You know every place the documents live. At Acme the supply agreements were spread across three SharePoint sites, one per region.
Readable. A program can extract the text. A scanned PDF is a picture of a page, so it needs OCR (optical character recognition, software that turns images of text into text) before anything else can happen.
Current. There is one version that counts. Two Northmart agreements, v3 and v4, were both in the library. An agent that quotes v3 is confidently wrong.
Owned. Someone is responsible for the content. The commercial team owns contract terms, so they, and not the FDE, decide which version is authoritative.
Permitted. You know who may see it. Pricing terms are for the commercial team only, and the agent must keep it that way.

In my work with partners, most “the AI is wrong” escalations end at one of these five checks.

What is RAG, in plain words?

A large language model only knows what was in its training data. It has never seen Acme’s contracts. RAG fixes that at question time: before the model answers, your code looks up the most relevant passages from your own documents and puts them into the prompt with the question. The model then writes its answer from those passages.

The formal definition comes from the 2020 paper that named the technique: RAG combines a model’s parametric memory (what it learned in training, stored in its weights) with a non-parametric memory (an external index of documents that is searched at question time). Broken into parts, that gives you a vocabulary you will use every day:

Corpus. The full set of documents you want to answer from.
Chunk. A small piece of a document, such as a section or a few paragraphs. You retrieve chunks, never whole documents.
Embedding. A list of numbers that represents the meaning of a chunk.
Index or vector store. The database that holds the chunks and their embeddings, built so that it can find similar ones quickly.
Retrieval. Finding the chunks most relevant to a question.
Grounding. Making the answer depend on the retrieved chunks rather than on the model’s memory.
Citation. A link from a sentence in the answer back to the chunk that supports it.

One everyday comparison helps. RAG turns a closed-book exam into an open-book exam. The corpus is the textbook, the index is its table of contents, retrieval is finding the right pages, and generation is writing the answer with those pages open. Finding the right page is where most of the engineering goes.

RAG has two halves that run at different times. Watch a document go through the first half, then a question go through the second:

the two halves of RAG: an indexing pipeline that runs on a schedule turns documents into an index, and a query pipeline that runs on every question retrieves the best chunks and gives them to the model with the question

How do documents become chunks?

Parsing is turning a file into clean text with its structure. Plain text extraction from a PDF loses headings and flattens tables into a stream of numbers. A layout-aware parser keeps the structure: which line is a heading, which cells belong to which row. On Google Cloud, the Document AI layout parser does this, and RAG Engine can use it during ingestion.

Chunking is splitting the parsed text into pieces small enough to retrieve precisely. There are three common strategies:

Fixed size. Every N tokens, usually with some overlap (the end of one chunk repeated at the start of the next). Simple and predictable. RAG Engine’s defaults are 1,024 tokens per chunk with 256 tokens of overlap.
Structure-aware. Split on headings and keep tables whole. Better for contracts, policies and manuals, where a section is a natural unit of meaning.
Semantic. Split where the topic changes, using embeddings to detect the change. Useful for long, unstructured text such as transcripts.

This is what fixed-size chunking did to a Northmart penalty table, and what a structure-aware chunk looks like instead:

chunking one contract page two ways: fixed-size chunks cut the penalty table in half, while a structure-aware chunk keeps the whole section together and carries metadata such as source, version and who may see it

Two lessons came out of this at Acme. First, choose the chunking strategy per document type, not once for the whole corpus. Second, attach metadata (facts about the chunk, such as source, page, version and allowed groups) at ingestion time. Metadata is what lets you filter out v3, enforce permissions and cite a page number later. It is far harder to add afterwards.

What is an embedding, and how does vector search use it?

An embedding model turns text into a vector, a fixed-length list of numbers, in such a way that texts with similar meaning get similar vectors. On Google Cloud, gemini-embedding-001 returns 3,072 numbers by default, and the output_dimensionality parameter lets you ask for fewer to save storage. The API also takes a task type: you embed chunks with RETRIEVAL_DOCUMENT and questions with RETRIEVAL_QUERY, so each side is optimised for its role.

embeddings place text with similar meaning close together: the question about late delivery fines lands next to the penalty clauses, and vector search returns its nearest neighbours by cosine similarity

Vector search embeds the question and returns the chunks whose vectors are closest, usually measured by cosine similarity (how closely two vectors point in the same direction). Comparing the question with every chunk is called exact or k-nearest neighbour (kNN) search. It is accurate but slows down as the corpus grows. Approximate nearest neighbour (ANN) search uses a smarter index to check only a small fraction of the vectors, trading a little accuracy for a lot of speed. Google’s Vector Search is built on ScaNN, an ANN algorithm from Google Research.

Why is vector search alone not enough?

Embeddings are good at meaning and weak at exact strings. Ask for “clause 14.2” or the product code “PL-500” and a vector search may return chunks that are about the right topic but quote the wrong clause. Keyword search (also called sparse or token-based search, often scored with an algorithm such as BM25) is the opposite: precise on exact words, blind to synonyms.

Hybrid search runs both and merges the two ranked lists. Vector Search on Google Cloud supports this directly and merges results with reciprocal rank fusion (RRF), which rewards a chunk for ranking high in either list. Then a reranker reads the question and each shortlisted chunk together and scores how well the chunk answers the question. The ranking API on Google Cloud does this, and the RAG Engine documentation describes its latency as very low, under 100 milliseconds.

the retrieval funnel: keyword search and vector search each return about fifty candidates, reciprocal rank fusion merges them, a ranking model reorders them, and only the best five reach the model

Cheap, wide searches find candidates; the precise step only runs on the shortlist. At Acme, keyword search fixed the questions that quoted clause numbers, and reranking moved the right Northmart section to first place.

How do grounding and citations keep the answer honest?

Retrieval gives the model good material. It does not force the model to use it. Grounding is the discipline of checking that every claim in the answer is supported by a retrieved chunk, and citations make that support visible to the reader.

Google Cloud offers a check grounding API for exactly this. You send an answer candidate and the facts it should rest on, and it returns a support score from 0 to 1 plus citations for each claim. You choose a threshold below which the answer is not shown as it is.

grounding an answer: each claim in the draft answer is linked to the chunk that supports it; one claim has no support, so the support score falls below the threshold and the claim is removed before the user sees it

The agents series applies the same rule to numbers, where code checks every figure the writer quotes (Part 5: The router). The model writes, and something other than the model decides whether the answer can be shown.

Which managed options does Google Cloud offer?

You rarely need to build all of this yourself. As of October 2026, there are four levels to choose from, and a renaming to be aware of: the product formerly called Vertex AI Search is being renamed Agent Search, and the former Vertex AI platform is now Gemini Enterprise Agent Platform. Older blog posts and code samples still use the old names, and the API endpoint (discoveryengine.googleapis.com) has not changed.

four ways to build retrieval on Google Cloud, from least to build to most control: Gemini Enterprise data stores, Agent Search, RAG Engine, and your own pipeline from embeddings, Vector Search, the ranking API and the check grounding API

Gemini Enterprise data stores. Connectors that bring SharePoint, Google Drive, Box and other sources into Gemini Enterprise with the source system’s permissions. Choose it when the users are employees working in the Gemini Enterprise app. Part 5 covers connectors in depth.
Agent Search. Managed ingestion, parsing, chunking, search, ranking and answers behind an API. Choose it when your own application or agent needs search or grounding, such as a contract search tool for the supply agent, and you want Google to run the pipeline.
RAG Engine. A managed RAG service built around a corpus, where you choose the chunk size, the parser and the vector database (its own managed database by default, Vector Search, Pinecone, or Weaviate in Preview). Choose it when you want control over how documents are prepared.
Your own pipeline. The building blocks used separately: the embeddings API, Vector Search, the ranking API and the check grounding API. Choose it when your data or scale is unusual, or retrieval quality is your product.

One honest gap: as of October 2026 the RAG Engine documentation says new projects must be added to an allowlist to use it in us-central1, us-east1, us-east4 and europe-west1, and several other regions are still in Preview. Check the region table before you promise a design.

My view: start at the left and move right only when an evaluation shows a retrieval problem the simpler option cannot fix. Every step to the right is more code the customer has to own after you leave.

How do I know retrieval is working?

Evaluate retrieval separately from the answer. If you only score final answers, you cannot tell whether a wrong answer came from a bad search or a bad generation, and you will tune the wrong thing.

The method is simple. With the data owners, write 30 to 50 real questions, and for each one note the chunk that should answer it. Run your retriever and record what it returns. Then compute two numbers:

Recall@k. The share of questions where the right chunk appears in the top k results. If it is not retrieved, the model cannot use it.
Mean reciprocal rank (MRR). The average of 1 divided by the position of the right chunk (1 for first place, 0.5 for second, 0 if missing). It tells you how high the right chunk sits.

Only once retrieval is good do you score the answers themselves, for correctness and groundedness, which is what Part 10 of the agents series builds into a release gate.

Why do RAG answers go wrong?

When a customer shows me a wrong answer, I walk this tree with them. It turns “the AI is wrong” into a specific, fixable cause:

diagnosing a wrong RAG answer: first ask whether the right chunk was retrieved; if not, check whether it is in the index at all, then the ranking; if it was retrieved, check whether the answer followed it

Hallucination. The right chunk was retrieved, or nothing relevant was, and the model wrote something plausible anyway. Fix it with a grounding check and an instruction to say “I could not find that” when support is weak.
Stale data. The index is older than the source. Every connector and ingestion job has a sync schedule; a contract signed this morning is invisible until the next sync. Show the document date in every citation so users can see it.
Permissions. This is the one that matters most in an enterprise. If you ingest documents with a single service account, the index no longer knows who was allowed to read each file, and every user of the agent can now see everything that account could see. RAG Engine’s Google Drive import, for example, works by sharing the folder with a service account. Gemini Enterprise connectors, by contrast, check the signed-in user against the source system’s access control lists. Decide which behaviour you need before you ingest anything.
Bad parsing or chunking. The table was flattened or cut in half, as in the Northmart example. Look at the chunks themselves, not only at the scores.

Try it

This script needs nothing but Python. It scores a small, made-up set of Acme retrieval results and prints recall and MRR. Change the got lists to see how the numbers move.

"""Measure retrieval before you measure answers: recall@k and MRR."""

# Each case: a question, the chunk that should answer it,
# and the chunk IDs the retriever returned, best first.
results = [
    {"q": "Northmart penalty below 90% OTIF?", "expected": "northmart_v4_p7_s14.2",
     "got": ["northmart_v4_p7_s14.2", "northmart_v3_p7_s14.2", "sop_p2"]},
    {"q": "Who books carrier slots?", "expected": "sop_p2",
     "got": ["sop_p4", "sop_p2", "spec_pl500_p1"]},
    {"q": "Shelf life of Acme Pure 500ml?", "expected": "spec_pl500_p1",
     "got": ["spec_pl500_p2", "spec_pl250_p1", "sop_p2"]},
    {"q": "Notice period to end the Northmart deal?", "expected": "northmart_v4_p19_s22",
     "got": ["northmart_v3_p18_s22", "northmart_v4_p19_s22", "northmart_v4_p7_s14.2"]},
]

def recall_at_k(rows, k):
    hits = sum(1 for r in rows if r["expected"] in r["got"][:k])
    return hits / len(rows)

def mrr(rows):
    total = 0.0
    for r in rows:
        if r["expected"] in r["got"]:
            total += 1 / (r["got"].index(r["expected"]) + 1)
    return total / len(rows)

print(f"recall@1 = {recall_at_k(results, 1):.2f}")
print(f"recall@3 = {recall_at_k(results, 3):.2f}")
print(f"MRR      = {mrr(results):.2f}")
for r in results:
    if r["expected"] not in r["got"]:
        print("MISSED:", r["q"])

You should see recall@1 = 0.25, recall@3 = 0.75 and MRR = 0.50, with the shelf-life question missed. Then try two things:

Look at the last case. The old v3 contract ranks above v4. Remove northmart_v3_p18_s22 from the list, which is what a metadata filter on version would do, and watch MRR rise.
Write five questions about documents you know well, with the chunk that should answer each. That is the start of a real retrieval test set.

What can go wrong in the engagement

Promising accuracy before measuring it. Agree the test set and a target first.
Ingesting everything. More documents means more near-duplicates and stale versions. Start with what the first use case needs.
Treating the FDE as the data owner. You can find a stale contract; only the commercial team can retire it. Name an owner for each source in the engagement brief from Part 2.
Tuning the prompt to fix a retrieval problem. If recall is low, no prompt will help.

What I would do now

Run the five readiness checks on the first use case’s documents and write down every failure.
Parse ten representative documents and read the chunks yourself before indexing the rest.
Attach source, page, version, owner and permission metadata at ingestion.
Build a 30-question retrieval test set with the data owners and track recall@5 and MRR.
Start with the most managed option that fits, and add hybrid search, reranking or a grounding check only when the numbers say you need them.

What did we learn?

Data is ready when an agent can find it, read it, trust that it is current, know who owns it and respect who may see it.
RAG looks up relevant passages from your own documents at question time and gives them to the model with the question, so the answer can cite where it came from.
The chunking strategy should be chosen per document type, and metadata such as source, version and permitted groups must be attached at ingestion.
Vector search is strong on meaning and weak on exact strings, so hybrid search followed by reranking usually retrieves better than either search alone.
Google Cloud offers four levels, from Gemini Enterprise data stores to your own pipeline, and it pays to start with the most managed option that fits.
Retrieval should be measured with recall@k and MRR before the answers are scored, and a wrong answer should be traced through the diagnosis tree to a specific cause.

What comes next?

Retrieval is one kind of integration. In Part 5: Enterprise integrations, APIs, connectors and identity, we connect the agent to the rest of Acme’s estate: REST APIs and how they authenticate, the SharePoint, Box and Google Drive connectors in Gemini Enterprise, the identity standards (SAML, OIDC and SCIM) that keep permissions intact, and why MCP servers are becoming the integration layer.

Sources

Opinions are my own. Every fact in this post comes from the public documentation linked above. Acme Consumer Brands and its data are fictional.