Skip to main content

How to et "Enterprise RAG Experience" Without the Enterprise Job

Open any AI engineer or Forward Deployed Engineer job post and you’ll see some version of the same line: “RAG experience on real enterprise data.”

Updated 4 min readMarkdown
How to et "Enterprise RAG Experience" Without the Enterprise Job

Open any AI engineer or Forward Deployed Engineer job post and you’ll see some version of the same line: “RAG experience on real enterprise data.”

That leaves most people stuck. How do you get enterprise experience without already working at an enterprise?

This week’s answer is a dataset I keep coming back to, plus the lab I built around it.

Meet TechQA

TechQA is a free dataset of real IBM technical support questions, asked by real IBM customers. It isn’t synthetic and it isn’t tidy Wikipedia trivia.

Every question comes with 50 candidate documents (IBM “technotes”):

  • One is the right page.
  • The other 49 are look-alikes: same product, same error codes, almost the same title.
  • And sometimes none of them are right.

That last point is the important one. It’s exactly what a client’s knowledge base looks like: messy pages that all sound the same, and questions where the honest answer is “I can’t find support for that.”

Learn to handle that and you’ve learned what clients actually pay for.

The lesson you can’t fake

Here’s what separates people who’ve built this from people who’ve watched videos about it.

TechQA tells you where each answer sits inside its source page (a start and end offset). So you can measure something most tutorials never check.

When I measured it, 16% of answers start after character 2,000.

Now look at what a lot of tutorials do:

text = doc[:2000]   # “just keep the opening of each document”

For that 16%, the answer is gone before search even begins. No retriever, re-ranker or prompt can recover text you’ve already thrown away.

You only learn this by measuring.

What you’ll build in the RAG Lab

The RAG Lab home page: a left sidebar listing the course modules and a main panel with the title, an intro, and cards for each level

The lab takes you from raw data to a measured baseline:

  1. Inspect the data: splits, answerable vs. unanswerable questions, candidate lists.
  2. Compare raw HTML with clean text: what does stripping actually change?
  3. Measure answer positions: where do answers live, and what does truncation cost you?
  4. Build a BM25 search: a simple, honest keyword baseline.

On 160 real questions, the BM25 baseline found:

  • the right page first 47% of the time
  • the right page in the top 5 63% of the time

That’s your number to beat. Even better, the failures tell you what to fix next: the query, the documents, or the ranking.

That’s the exact workflow you’d walk an interviewer through: data first, then measurement, then a baseline, then targeted fixes.

Real scale, on your laptop

You start with about 28,500 documents, then move to the full library of 802,000. That’s 28× bigger.

At that size you hit the problems you’d normally only meet on the job: indexing time, memory, and whether your results still hold up as the data grows.

Try this before next week

Even if you never open the lab, try this one exercise on whatever RAG project you have:

Find out where your answers live. For a handful of questions you know the answers to, record the character position of the answer in its source document. Then check whether your chunking or truncation keeps it.

If you find answers past your cut-off, you’ve just found a bug that no amount of prompt engineering will fix.

What’s next

This is Part 1 of a 3-part series on RAG for FDEs: retrieval, or finding the right source before you ask a model to answer.

You don’t need an enterprise job to build enterprise skills. You need the right dataset, honest measurement, and a system that knows when to say “I can’t find support for that.”

https://www.youtube.com/watch?v=74K4UN4M7cw

Hit reply and tell me your BM25 numbers (or where your answers ended up living). I read every reply.

See you in Part 2, Zahir The Adaptive Engineer

About the author

  • I am a Journalist-turned-Software Engineer. I love coding and the associated grind of learning every day. A firm believer in social learning, I owe my dev career to all the tech content creators I have learned from. This is my contribution back to the community.

No comments yet. Be the first to comment.