Bonbonnetje is an AI Agent running in production helping Xebians submit expense claims. The Agent is built with Agent Development Kit and is running on Google Cloud Platform. Bonbonnetje is in active use by multiple internal departments and successfully completes hundreds of submissions monthly, helping save both employees and back-office valuable time.

Why an Agent for expense claims?

Why, you might ask - create an Agent to help you submit your expense claims? Let us explain:

• Less effort. Any task you can do in under two minutes, do it immediately. With Bonbonnetje, we reduce the time spent on processing your expenses to less than a minute. This way, you don't need to add more todo items to your todo list - because we understand you - it is already long enough.
• Reduced back-office stress. Our Agent performs policy checks and notifies the employee before they submit. This helps reduce submission errors before they even reach our back-office Finance team, saving time.
• Cost saving. Time is money. Bonbonnetje helps save precious time for 1) Xebia employees and 2) Xebia back-office alike. This implies a significant yearly cost saving.
• Learning. What's a better way to learn than to actually build and deploy? Building- and maintaining production Agents is hard- which is the exact thing making it a good practice for our team and beyond.
• Authority. At Xebia we are shipping production Agents for clients. It is straightforward, then, to build internal Agents too. So, let's practice what we preach and Show Our Work! 😉

... and, besides the above, maybe even the most important reason of all: making submitting expenses more fun! 🍬 How valuable is that, to take an otherwise tedious process in your workweek and turn that into something more enjoyable?

Building an Agent that runs and keeps running

Running Agents in production in a reliable, available and observable fashion is not trivial. It requires one to think on how to do session- and state management, artifact storage, Agent observability and Agent evaluation. We choose Gemini Enterprise Agent Platform for this, formerly known as Vertex AI Agent Engine. Our setup can be visualised as follows:

Let's dive into several components in more detail.

An Agent in production: the necessary ingredients

To build Agents several ingredients are needed. We need to define tools, track user conversations in sessions which each have a state, be able to handle artifacts like images, PDFs and other files, and we need to stream responses to the user. That needs to be powered by a model we can pull in, like Gemini 3.1 Flash Lite. That needs to execute in a loop of our choosing and be able to tap into Sub-Agents we define, an orchestrator if you will. We need to be able to plug in custom callbacks into tool calls, LLM calls and other lifecycle events happening, to control the full flow of the Agent. I think you get the gist by now - this can be quite a lot to manage.

That's where an Agentic framework comes in - an Agentic framework can help us build faster by providing us reusable components. There are many Agentic frameworks by now. We chose Google's Agent Development Kit (ADK) for this use case. ADK has concepts of sessions, state, memory, artifacts and more. It allows for easy local development with a web interface you can spin up by running adk web. But, most importantly, we are creating a production Agent here. That means we need for each of those concepts a scalable solution that can run Cloud-native in a stable manner. We also need proper observability around all our Agent-interactions.

This is where ADK maps towards Gemini Enterprise Agent Platform (formerly Vertex AI Agent Engine), allowing you to consume a SessionService, ArtifactService and an observability stack consisting of Cloud Trace, Cloud Logging and Cloud Monitoring. This is what the stack and the mapping towards the managed offering look like:

Awesome! We now have the core of the Agent set. To make things operational we do need some Ops. We are using GitHub Actions as a driver for our CI/CD needs, combined with Terraform to spin up relevant Cloud resources:

Tip: the google/agents-cli skills have been very useful in this process.

Agent Observability

Agents are far from deterministic and having proper instrumentation in place so your Agent can be monitored is key. Gemini Enterprise Agent Platform integrates tightly with offerings like Cloud Trace and Cloud Logging. Specifically Cloud Trace comes in very handy, allowing us to instrument all Agent interactions happening in a session. Tool calls, Sub-Agent handoffs, latencies and such. The essential concepts here are traces and spans- where a trace is made up of multiple spans. We can visualise our instrumentation and the managed offerings for observability they target as follows:

Please observe in the diagram OTel. This is OpenTelemetry- which conveniently integrates with Cloud Trace. One such trace can look like the following:

... which visualises, in a trace/span view, one typical Agent interaction with Bonbonnetje. This allows for pinning down latency bottlenecks and easy debugging.

Concluding

Bonbonnetje is an Agent built to help Xebians both save time in submitting their expenses and making it more fun 🍬. Because the Agent also feeds back to the user policies that apply to the expense, errors are caught earlier in the ERP pipeline making for back-office time savings too. Bonbonnetje is running stably in production for several months on Gemini Enterprise Agent Platform and was built with Agent Development Kit with an AI team at Xebia Data.

Important aspects of running an Agent in production are an Agent framework, scalable solutions for session-, state- and artifact management, and proper observability mechanics.

Having built something techy that actually saves time makes us very happy. But more than anything- it was also awesome to build. Happy days!

Core team: Jeroen Overschie, Victor de Oliveira, Jordi Smit, Sinan Çalışır, thanking our Finance team Susanne van Santen, Koen van Laarhoven, Donna van Ek-Grob, Mark Jansen for the great collaboration and thankful for contributions made by Nelli Gofman, Roel Bertens, Jacopo Pierotti, Roy van Santen, Bing Tan, Julian de Ruiter, Yannick Bosch.

Large Language Models (LLMs) are impressive generalists. They perform increasingly well on a wide range of tasks; achieving human-level or even superhuman-level performance on many benchmarks. However, being good at everything can come at a cost. These so-called Foundation models have billions of parameters and require a vast amount of GPUs to train- or run. Unaffordable for most - so we consume LLMs from APIs instead - which comes with its own considerations like cost, privacy, latency or vendor lock-in. This is not the only option. When only specialist capabilities are required, generalist capabilities can be sacrificed in favor of domain-specific knowledge, allowing us to get away with smaller models. Smaller models require less compute and can achieve similar- or even better performance than larger models on specific tasks ^[1]. With enough data available, we can specialise existing models ourselves. This is fine-tuning.

What is fine-tuning?

Fine-tuning is a process where we take a pre-trained model and continue training it on a new dataset, typically with a smaller learning rate. This allows the model to adapt to the specific characteristics of the new data while retaining the general knowledge it gained during the initial training. In practice, this often involves further training the top layers of the model while freezing the lower layers, which helps to preserve the learned features from the original training.

So, how to finetune ourselves? Let's figure that out! 

How to finetune yourself ✓✓

Enough talking! Let's get practical. Let's fine-tune a model on our own hardware. We have access to the following setup:

• HP Z8 Fury
• 3x NVIDIA RTX 6000 Ada GPUs

Our goal is: to finetune a model and benchmark it with MMLU Pro, math category.

So in order to start fine-tuning, we need to decide on the following:

1. Fine-tuning framework: Axolotl

There are several fine-tuning frameworks: Axolotl, Unsloth and torchtune. We choose Axolotl for its ease of use and its rich out-of-the-box examples.

In Axolotl's first fine-tune example, you only need to execute a one-line command to fine-tune a Llama3.2 model:

axolotl train examples/llama-3/lora-1b.yml

The fine-tuning dataset and method are all defined in the YAML file.

2. Model choice

The first question we asked is how large a model can we fine-tune given 3 GPUs. One NVIDIA RTX 6000 Ada GPU has 48 GB, so in total we have around 150 GB.

Under the hood, Accelerate is used. We need to figure out, however, how big of a model we can train on the available GPUs. To calculate the memory usage based the amount of model parameters, Zachary Mueller's slides offer a convenient formula.

Take Llama 3 8B model for example, which has 8 billion parameters. Each parameter is 4 bytes, so the model needs roughly 8 × 4 GB. Further, the backward pass take 2× the model size, and optimizer step takes 4× the model size, which consumes the highest memory during fine-tuning.

You can also use this model size estimator tool to estimate the memory usage based on your own model choice.

For example, we cannot fine-tune a DeepSeek V2 model, because the memory usage is above 3 TB, as shown below:

Based on the memory limit, we choose Gemma-2-2B to fine-tune and dip into the fine-tuning techniques.

The reason we choose a Gemma model is that we want to use a relatively small model to fine-tune for a specific task to prove that a fine-tuned smaller model can outperform the larger models in a specific domain.

3. Benchmarking dataset: MMLU-Pro

For the benchmarking dataset, we use the MMLU-Pro dataset. It has 14 categories of questions, ranging from math, physics, biology to business and engineering. For each question, there are 10 options to choose from for one correct answer. When it comes to the dataset split, it has 70 rows for validation and 12k rows for test. The validation dataset is used to generate CoT (Chain of Thought) prompts for the test dataset. Together, they form the prompts to benchmark your model.

The MMLU-Pro leaderboard shows the benchmarking results for an abundance of models. For example:

Since only the overall accuracy is reported for Gemma 2- or 3- models, we also want to know the specific accuracy for math.

The benchmarking script can be found in MMLU-Pro's GitHub repository. For example, if we want to evaluate a local model, we can reference this script:

• model is <repo>/<model-name> from Hugging Face.
• selected_subjects can be all for all categories, or math for a specific category.

python evaluate_from_local.py \
                 --selected_subjects $selected_subjects \
                 --save_dir $save_dir \
                 --model $model \
                 --global_record_file $global_record_file \
                 --gpu_util $gpu_util

We ran the above script for --selected_subjects math, we get the below results:

Next, we want to fine-tune a smaller Gemma model to improve the accuracy of math above 0.107.

4. Training dataset: AceReason-Math

We use NVIDIA's AceReason-Math dataset to fine-tune our model. It is a training dataset with 49.6k math reasoning questions. Instead of giving a list of options, the dataset comes in the format of one question and one correct answer.

In order to use this question/answer format to fine-tune a model in Axolotl, we need to check what instruction tuning formats are provided. This alpaca_chat.load_qa format perfectly matches our training dataset. So we can update our fine-tuning YAML file as below:

datasets:
  - path: /data/nvidia_ace_reason_math.parquet
    type: alpaca_chat.load_qa

One thing to pay attention to is that we also need to match the dataset column names to question and answer, so we rename the original problem column to question.

5. Fine-tuning strategies: QLoRA, LoRA, FSDP, DeepSpeed

There are many techniques and libraries around that help make the fine-tuning process more efficient, many of which focussing on memory efficiency and performance. Most notably, these include LoRA, QLoRA, FSDPand DeepSpeed. Axolotl supports all of these techniques. DeepSpeed or FSDP can be used for Multi-GPU setups, of which FSDP can be combined with QLoRA.

So what are these techniques?

LoRA (Low-Rank Adaptation) is a technique that allows us to fine-tune large models with fewer parameters by introducing low-rank matrices.

LoRA ... [reduces] trainable parameters by about 90%https://huggingface.co/learn/llm-course/en/chapter11/4

... this means, that we can fit more in memory and more easily fine-tune on consumer-grade hardware. Awesome!

• QLoRA (Quantized LoRA) is an extension of LoRA that quantizes the model weights to reduce memory usage further.
• DeepSpeed is a library that helps you do distributed training and inference supported by Axolotl.

FSDP (Fully Sharded Data Parallel) is a way of distributing work over GPUs.

FSDP saves more memory because it doesn’t replicate a model on each GPU. It shards the models parameters, gradients and optimizer states across GPUs. Each model shard processes a portion of the data and the results are synchronized to speed up training. - FullyShardedDataParallel (HuggingFace)

Combinations of LoRA/QLoRA are used as an adapter in the Axolotl config in combination with FSDP for finetuning Gemma2/Llama3.1/Qwen2 on our limited amount of GPU resources ✓. Let's now see how to run the fine-tuning process! 🚀

6. Running the fine-tuning 🚀

Axolotl provides a list of Docker images for us to use out of the box. In order to use it, you can reference this Docker installation.

Since we have our development GPU infrastructure set up and managed by Argo, we create a Kubernetes job to run the fine-tuning process as below:

apiVersion: batch/v1
kind: Job
metadata:
  name: axolotl
spec:
  backoffLimit: 0
  template:
    metadata:
      labels:
        app: axolotl
    spec:
      restartPolicy: Never
      containers:
        - name: axolotl
          image: axolotlai/axolotl:main-latest
          command: ["/bin/bash", "-c"]
          args:
            - |
              set -e
              echo "Start training..."
              axolotl train $YAML_FILE --num-processes $NUM_PROCESSES
              echo "Merging the LoRA adapters into the base model..."
              axolotl merge-lora $YAML_FILE --lora-model-dir=./outputs/out/
              echo "Uploading the fine-tuned model to Hugging Face..."
              huggingface-cli upload $MODEL_NAME ./outputs/out/merged/ .
          securityContext:
            privileged: true
            runAsUser: 0
            capabilities:
              add: ["SYS_ADMIN"]
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: "40Gi"
            requests:
              memory: "20Gi"
          volumeMounts:
            - name: axolotl
              mountPath: /workspace/axolotl
            - name: huggingface
              mountPath: /root/.cache/huggingface
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: axolotl
          persistentVolumeClaim:
            claimName: axolotl-pvc
        - name: huggingface
          persistentVolumeClaim:
            claimName: huggingface-pvc
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: "10Gi"

The variables come from the config map defined in kustomization.yaml:

• NUM_PROCESSES is the number of GPUs to use
• YAML_FILE is the YAML file from examples
• MODEL_NAME is the model name to upload to Hugging Face.

Results

After the fine-tuning is done, which takes roughly 2.8 hours, we can use python evaluate_from_local.py to evaluate the result, which is as below:

Compared to a non-fine-tuned Gemma-3-12B-it model based on math, the fine-tuned model improves the accuracy of math from 0.107 to 0.238 ✓.

Conclusion

In this blogpost we learned: - What fine-tuning is and why it is useful. - How to fine-tune a model on our own hardware using Axolotl. - How to benchmark the fine-tuned model using MMLU-Pro. - How to run the fine-tuning process in a Kubernetes job.

We successfully fine-tuned a model and benchmarked it with MMLU-Pro, math category. The results show that fine-tuning smaller models can lead to competitive performance on specific tasks, demonstrating the power of domain-specific knowledge in LLMs.

Finetuning is definitely not always the right approach - but if it is - it can be very powerful. Good luck in your own fine-tuning adventures! 🍀

Citations

1. Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., ... & Natarajan, V. (2025). Toward expert-level medical question answering with large language models. Nature Medicine, 31(3), 943-950.

uv is the new king in Python package manager land. It's fast, comprehensive and by now, well-adopted. Devcontainers provide a powerful way to create a reproducible development environment, by developing inside Docker containers. So, how do we combine the two? To find out, read along.

How to create a Python + uv Devcontainer

Let's set up our project step-by-step.

Step 1: Devcontainer.json

1. Create devcontainer.json
   This is the Devcontainer specification telling an editor everything it needs to know about our Devcontainer. Most importantly, we decide which Docker image is to be used. There are official Python images that available on Docker Hub, like python:3.13. However, this image is not meant for development. It is a minimal image used to run Python applications. We don't get things like Zsh, Git or a non-root user. For this reason, there are pre-built Devcontainer images we can use. We will use mcr.microsoft.com/devcontainers/python:3.13:

   {
       "image: "mcr.microsoft.com/devcontainers/python:3.13"
   }
   

   ... great!

2. Adding uv
   To add uv, we can use Devcontainer Features:
   
   

   
   
   

   Devcontainer Features are pieces of installation code which can be added to your Devcontainer in modular fashion. They are shareable and there is a large collection of features available for us to use. So is the case for uv! There is a Devcontainer feature for uv. We can add uv like so:

   {
       "image: "mcr.microsoft.com/devcontainers/python:3.13",
   
       "features": {
           "ghcr.io/jsburckhardt/devcontainer-features/uv:1": {},
           "ghcr.io/jsburckhardt/devcontainer-features/ruff:1": {}
       },
   }
   

   ... great! We also added a Devcontainer feature for Ruff. Ruff is a fast and widely used tool for Python- linting and formatting. The latest versions of uv and Ruff are installed unless specified otherwise.

3. Adding VSCode customizations
   Although Devcontainers are supported by various editors, VSCode is the most common. GitHub Codespaces also uses VSCode, meaning we get full Devcontainer support in GitHub Codespaces too.

   We can define extensions to be installed automatically alongside our Devcontainer, by defining customizations:

   {
       "image": "mcr.microsoft.com/devcontainers/python:3.13",
   
       "features": {
           "ghcr.io/jsburckhardt/devcontainer-features/uv:1": {},
           "ghcr.io/jsburckhardt/devcontainer-features/ruff:1": {}
       },
   
       "customizations": {
           "vscode": {
               "extensions": [
                   "ms-python.python",
                   "ms-toolsai.jupyter"
               ]
           }
       }
   }
   

   ... the above installs the VSCode Python extension and Jupyter extension automatically when the Devcontainer is opened.

With only just our .devcontainer.json file, we can already start the Devcontainer! Install the VSCode Dev Containers Extension and find the blue popup saying Reopen in Container:

Upon opening our Devcontainer, the features we defined earlier are installed for us. Python, uv and ruff are all there:

Great ✓. Let's continue with our Python project setup.

Step 2: Python project setup with uv

Now that we have a Devcontainer, let's set up our project inside it.

1. uv init

   We can use uv init to scaffold a new Python project.

   uv init \
     --package \
     --name example_project \
     --description "Example Python Devcontainer project with uv"
   

   ... which scaffolds a project structure for us:

   

   Importantly, a pyproject.toml file is created. This is the main configuration file for our Python project. Besides configuration, it keeps track of dependencies. Let's add some dependencies.

2. Adding dependencies

   We want to test our code, so let's add pytest:

   uv add pytest --dev
   

   ... upon installing, VSCode also detects a virtual environment being created:

   

   Select the virtual environment. Notice that after the installation uv has updated our pyproject.toml file.

   💡

   If you missed the notification, open the command palette (CMD+SHIFT+P), type Python: Select Interpreter and select ./.venv/bin/python.

3. Running tests
   Let's now add a test so we can validate our setup is working.

   

   ... which we can run using:

   uv run pytest .
   

   Resulting in ... a passing test! ✓

   

   Also, feel free to use the VSCode UI:

   

   Awesome! 🎉

Final setup

At last, our project looks like the following:

For ease of use, this GitHub repository is available and can be used as a template:

GitHub - dunnkers/python-uv-devcontainer: Python project setup using a Devcontainer and uv.

Python project setup using a Devcontainer and uv. Contribute to dunnkers/python-uv-devcontainer development by creating an account on GitHub.

GitHubdunnkers

... the repository also comes with some Extras: a GitHub Actions workflow for CI/CD and a Dockerfile for production deployments. Enjoy!

Conclusion

That was an adventure. We set up a Devcontainer using a devcontainer.json file, we added uv and ruff by using Devcontainer features and set up our Python project using uv init, uv add and uv run.

uv is a powerful tool and combining it with a Devcontainer gives us an easy and reproducible project setup. Good luck with your own Python/uv Devcontainer setup! 🍀

MLOps has survived the hype cycle and is gaining in maturity. But are we looking at MLOps for answers for the right things? No matter how valuable MLOps can be for you, without proper building blocks in place MLOps cannot live up to its full potential. What are the prerequisites for MLOps? What parts of MLOps should you focus on? When should you even start thinking about MLOps, or when is ‘plain’ DevOps wiser to focus on first? Read along to learn more!

About being ready

So, what does it mean to be ready? Being ready means understanding why you need that technology and what it is. Then, we can start understanding what it takes to adopt it and especially: what prerequisites are required to do so. With all the prerequisities in place, we can say we are ready.

In short, being ready for MLOps means you understand:

1. Why adopt MLOps
2. What MLOps is
3. When adopt MLOps

... only then can you start thinking about how to adopt MLOps.

As an analogy, think of building a house. You need the right foundation in place first before you build higher. Each level needs to be built properly before you advance. If the right foundations are not in place, you can build however beautiful house on top of those foundations but still the building will not stand time.

To start adopting MLOps the prerequisites need to be in place before we start building on top of them. What are those prerequisites? Let's find out if you are ready, together.

Why MLOps?

Why bother with MLOps? Let's first explore what pain you might experience that MLOps can help with. First let's throw in a statistic. Gartner reported that on average only 54% of AI models move from pilot to production:

... that is not an awful lot. What a waste! We spent time trying to get models into production but we are not able to. Why is that? These days Data Science is not anymore a new domain by any means. The time when Hardvard Business Review posted the Data Scientist to be the "Sexiest Job of the 21st Century" is more than a decade ago ^[1]. In 2019 alone the Data Scientist job postings on Indeed rose by 256% ^[2]. Universities have been pumping out Data Science graduates in rapid pace and the Open Source community made ML technology easy to use and widely available. Both the tech and the skills are there:

So then let me re-iterate: why, still, are teams having troubles launching Machine Learning models into production? Big part of the reason lies in collaboration between teams. Even though we all wish for seamless transitions from the development phase towards production, Machine Learning- development and operation teams can have conflicting interests, making it difficult to collaborate. The development- and operations world differ in various aspects:

• Development ML teams are focused on innovation and speed
  • Dev ML teams have roles like Data Scientists, Data Engineers, Business owners.
  • Dev ML teams work agile and experiment rapidly using PoC's.
  • Dev ML teams work in Jupyter notebooks, Python, R, etc.
• Operations ML teams are focused on stability and reliability
  • Ops ML teams have roles like Platform Engineers, SRE's, DevOps Engineers, Software Engineers, IT Managers.
  • Ops ML teams work with strict roadmaps, make long-term plans and might need to be available for on-call support.
  • Ops ML teams work in CI/CD pipelines, Kubernetes, Docker, Java, Scala, etc.

... that does not make things easier. The expectation is that dev- and ops magically work well together. We can just 'hand over' the model to the operations team and they will take care of it. This is not the case, unfortunately.

Expectation:

... versus Reality:

... such handovers make development cycles unnecessarily long and makes it difficult to get feedback from production models back to the developers. To conclude, a lack of collaboration between development and operations causes three issues:

• ❌ Few models in production ... or production solutions prove unreliable.
• ❌ Long development cycles ... to create models, update models or add features.
• ❌ Lacking feedback ... on model performance and added value.

How to solve this? Enter MLOps.

What is MLOps?

MLOps can help development- and production teams work better together and more efficiently deploy Machine Learning models to production. MLOps makes the dev- and ops worlds more familiar with each other and aims to bring the worlds closer together.

The term has gained in popularity since 2018 ^[3] [4], when the Machine Learning had undergone massive growth. Fair to say, some years have passed by now and we have moved beyond the hype by now:

So what is MLOps comprised of? No longer is Machine Learning development only about training a ML model. We are concerned about many other things, too. Preprocessing, feature engineering, serving, scheduling and monitoring to name a few. We can map those to concrete methodologies and tooling that makes up the technical part of MLOps:

... some might already come familiar to you. So do they to major Cloud Providers. Cloud providers have answered the market need for better tooling in the Machine Learning space. In fact every major provider has a Machine Learning platform that helps you do MLOps:

That is massively useful. Not only is more tooling available to do MLOps, there also exist managed solutions that you can use out of the box. 

Let's recall the three issues we had with handovers between development- and operations teams. Remember them? We can now map them toward what MLOps promises to give us, helping us with these issues. 

... that sounds great! We have MLOps as a key methodology to reduce the handover and bring dev- and ops closer together. But what actually empowers and enables MLOps? What do we need to do successful MLOps? Did we any time before bring together Dev and Ops teams? In fact, we did! It's in the name! Enter DevOps.

DevOps in the mix

MLOps is largely inspired by DevOps ^[5]. DevOps has existed for longer and is more established and mature. Since 2007 DevOps has been a massively influential methodology in software development. Key is that development is not sequential but continuous, best illustrated using the DevOps lifecycle:

In DevOps key principles are ^[6]:

• Automation
• Collaboration & communication
• Continuously improving
• Focus on user needs

... so how does MLOps interplay with DevOps? DevOps came from the Software Development world and therefore deals with Code. In Machine Learning, however, important to take into account are also Data and the Model:

Taking into account automating operations related to all of the code, data and model is what makes MLOps different from DevOps. For example, when it comes to automation we continuously check data quality, train models and run inference to create guarantees the state of our Machine Learning system.

Doing so continuously means we loop in our Monitoring practices back into our Model Building step like so:

That's great. So again, what is this interplay between DevOps and MLOps? It is clear that MLOps lent a lot from DevOps. But so in what way do we need DevOps to do MLOps? The way we see it, is that you need to have DevOps in place before you can do MLOps. DevOps is a prerequisite for MLOps. It's like stacking building blocks on top of each other, creating a solid foundation before you build higher:

That is promising, isn't it? Well, it is, but there's also effort required on the organisational side. MLOps is not just about tech.

The MLOps twist

MLOps is about more than just tech. MLOps is about all of Technology, People and Processes. Getting MLOps right means have carefully considered all three:

What does that mean? That means that in the MLOps lifecycle we want the ML product team to operate over the entire of the ML lifecycle, owning the entire process:

... this does indeed mean that we need a diverse set of roles, together in one team. Data Scientists, Machine Learning Engineers, Data Engineers and such need to work together. Working together, the handover can be minimised and models can be built that are actually ready for production.

To facilitate such a ML product team, a ML platform team is used to enable them. In a central team, they can provide best-practices and tooling to the ML product team. Key is, though, that still the ML product teams own the process from development to operation:

... admittedly, this is all not easy. Drawing out organisational changes on paper is easier than actually doing it. It requires serious effort to challenge your existing setup and restructure your organisation. But ask yourself - is it worth it? Is there serious Machine Learning potential in your organisation that you want to take advantage of? If yes, go for it.

Concluding: when to adopt MLOps

We learned a lot. We should do MLOps to eliminate handovers between dev- and ops teams and bring Software Engineering operational efficiency to the Machine Learning World. MLOps does so by providing useful technologies and tooling, widely offered by major cloud providers. MLOps is largely inspired by DevOps, which is a prerequisite for MLOps. This allowed the industry to take advantage of well-matured DevOps concepts like automation, collaboration and continuous improvement. Only do MLOps if you have your DevOps practices in place. 

If you end up doing so, you might very well end up in a better place because of it. Embrace the organisational investment and get teams closer together. We wish you good luck with your journey - and would love to help you along the way.

• [1] https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century
• [2] https://www.hiringlab.org/2019/01/17/data-scientist-job-outlook/
• [3] https://trends.google.com/trends/explore?date=all&q=mlops
• [4] https://www2.deloitte.com/content/dam/Deloitte/global/Images/infographics/technologymediatelecommunications/gx-deloitte-tmt-2018-predictions-full-report.pdf
• [5] https://en.wikipedia.org/wiki/MLOps
• [6] https://about.gitlab.com/topics/devops/
• [7] https://ml-ops.org/content/crisp-ml
  

GenAI has been around for a little while and its capabilities are expanding quickly. We are experiencing peak-hype levels for AI and expectations are sky-high. But still, many companies fail to generate actual value with GenAI. Why is that? Why are we promised so much yet manage to get so little? What is still fantasy and what concrete potential exists? What should be automated and what should not? In this blogpost, we explore the GenAI automation potential that exists today for data extraction.

Together, we will learn about:

1. Why GenAI data extraction
2. The automation levels
3. The automation potential

Let’s start!

1. Why GenAI data extraction

Why exactly should we care about GenAI data extraction? Let us motivate this by looking at 4 example usecases in different domains and with various data types like text-, images-, documents- or audio.

• Insurance claims (text)
• Menu cards (images)
• Annual reports (PDF documents)
• Customer service calls (audio)

Insurance claims (text)

Imagine you are running an insurance company and your employees are tasked with processing insurance claims. Claims are often provided in a free-form format: email, phone call, chat, in short: unstructured data. Your employees will then need to process this information to handle the claim: update the internal databases, find similar claim cases, etc. Let’s take the case where we receive this information in free-form text like emails.

We instruct the LLM to give its answers in structured format ^[ref], so we can easily work with- and transform its output. To clear this entire email box with some 25 emails Gemini 2.0 Flash took some 15 seconds. That is if ran sequentially: if ran in parallel this amount of data process can be processed in a matter of seconds. If this were scaled to process a mailbox of 100k emails we would spend about $1.52. Not an awful lot given we can save some serious time here. We can also half this cost if we use Gemini’s Batch API instead. How long would it take a human to process 100k emails? What is their time worth?

Extracting client information from this email message took Gemini 2.0 Flash about 0.5 seconds and cost $0.000015.

Menu cards (images)

Different usecase. Imagine now you are running an online platform for food delivery. You want to digitise menu cards so the offering can be ingested in your own platform in standardised manner. You are staffing a number of employees to help you do this task. The menu cards arrive in image format. Can we speed up the menu conversion process?

Yes: we can speed this up. LLMs consider the image as a whole and are also able to accurately process text details in the image. Because we are using a generic model images in many contexts can be processed by the same model without having to fine-tune over specific situations.

Converting this menu card took Gemini 2.0 Flash about 10 seconds and cost $0.0005. How long would a human take?

Annual reports (PDF documents)

Imagine you are tasked with answering questions based on financial documents. You are to go through each question and reason on the answer based on information provided in the documents. Take annual statements, for example. Companies worldwide are obliged to publish these statements and company finances are to be audited. Going through such documents is time-consuming. Take an annual statement in PDF document format. Can we help employees formulate answers based on this document faster?

Yes. The document fits in the model context all at once, allowing the model to take a comprehensive view of the document and formulate sensible answers. In fact, LLMs like Google’s Gemini models take up to 2 million tokens in context at once ^[ref]. This allows for up to 3,000 PDF pages to be inserted into the prompt. PDF pages are processed as images by Gemini ^[ref].

Answering these questions took Gemini 2.0 Flash just under 1 minute and cost $0.015 . How long would a human take?

Customer service calls (audio)

Imagine you have a customer service department and are processing many calls on the daily. The customer service agents do their best to document and record useful information during the call but cannot possibly document all valuable information. Calls are temporarily recorded for analytical purposes. Calls could be listened back to figure out the customer problem, the root cause, the solution and the customer sentiment. This information is valuable to have because this is necessary feedback information to improve customer service in a targeted way. What issues were we most often unable to resolve? In what situations are our customers left unhappy?

The insights are valuable, but hard to get to. The data is hidden in an audio format.

Gemini 2.0 Flash processed this 5-minute customer service call in 3 seconds and spent $0.0083. That is faster than any human takes to listen- and process the call. LLMs can also accurately process audio, without the need for a transcription step first. Because the model is exposed to the raw audio file, more than just the words that are spoken can be taken into account. In what way is this customer talking? Is this customer happy or unhappy? Off-the-shelf LLMs can be used without modification and just by prompting to extract structured data from audio ^[ref].

Concluding on the 4 usecase examples, we learned the following.

• LLMs for data extraction: LLMs can be used to extract structured data from free-form formats like text, images, PDFs, and audio.
• Larger contexts: Growing LLM context windows allow processing of larger documents at once. This strengthens the applicability for data extraction using LLMs without the need for extra retrieval steps.
• Competitive pricing: Lower inference costs make more data extraction use cases feasible.
• GenAI data extraction ROI: LLMs can complete data extraction tasks faster than humans and at a lower cost. Albeit not at perfection, the goal is to create a system that is good enough to assist humans in their work and provide business value.

That is really cool. LLMs are becoming more capable and cheaper, making possible more usecases than before. So now what exactly is this data extraction? How can we use this to our advantage to automate business processes? What are the automation levels for data extraction?

2. The automation levels

To discover what is possible with GenAI data extraction, we will divide into 4 levels of increasing automation.

• Level 0: Manual
• Level 1: Assisted
• Level 2: Autopilot
• Level 3: Autonomous

Level 0: Manual work

No automation is applied. Human labor is required to extract data in the desired format. The extracted data is then consumed by a human user. All of the data extraction-, evaluation of the data quality and any actions to be done with the data are manual human processes.

Without automation, manual work is required to extract data in structured format. After extraction humans decide on what to do with the data.

We can do better. LLMs can be used to automate part of this process, helping the human in the Assistedautomation level.

Level 1: Assisted

In the assisted workflow, a LLM is used to extract useful data. This process we like to call Structured Data Extraction.

LLMs can be used for structured data extraction: extracting useful data from free-form documents in structured format.

That can already save a lot of time. Previously we needed to manually perform the tedious process of processing the document to formulate answers in the target format. In this level of automation, though, we are still targeting a human user as output. The human user is responsible for all of evaluating the LLM output and deciding what to do with the data. This gives the human control but also costs extra time. We can do better in Autopilot.

Level 2: Autopilot

In the autopilot workflow we go a step further. The extracted data is directly ingested in a Data Warehouse, opening up new automation possibilities. With the data in the data warehouse the data can be more efficiently taken advantage of. We can now use the data for dashboards and insights as well as use the data for Machine Learning usecases.

Ingesting this data directly into a Data Warehouse can be beneficial indeed. But with the introduction of this automated ingestion we also need to be careful. Data Warehouse consumers are farther away from the extraction process and are not aware of the data quality. We need to always ask ourselves: Is this data reliable?

Bad data quality can lead to misleading insights and bad decisions later down. This is not what we want! Quantified metrics informing us on the data reliability are required. Those with expertise in the dataset and those involved in the extraction process will need to help out. These are typically an AI Engineer and a Subject Matter Expert. We need to introduce an evaluation step.

More steps are introduced, keeping the human-in-the-loop. The steps are necessary, though. The Subject Matter Expert plays a key role in ensuring the quality- and reliability of the data. This gives Data Warehouse consumers the trust they are looking for and at the same time enables the AI Engineer to more systematically improve the Structured Data Extraction system. Additionally, enabling the Subject Matter Expert themselves to be part of the prompting process lowers iteration times and reduces context being lost in translation between AI Engineer and Subject Matter Expert. Win-win.

We can go more automated, even. Let’s continue to the last level: Autonomous.

Level 3: Autonomous

In this level, the last human interactions are ought to be automated. Evaluation previously manually done by the Subject Matter Expert is now done by Quality Assurance- and evaluation tooling.

So what do we mean with such tooling? We want tooling to help us guarantee the outcome and quality of our data without minimal human intervention. This can be LLM-as-a-judge systems, for example. The required effort, however, to successfully implement such systems and run them safely is high. One gets value in return, from extra automation, but whether this is worth the effort- and cost is the question. Let’s compare the automation levels and summarise its potential.

3. The automation potential

We have learned about each of the four different levels of automation for data extraction using GenAI. Also, for each level, we have explored how architecturally a Structured Data Extraction system would look like. That is great, but now where is most potential? Let’s first summarise the automation levels as follows:

So say you are thinking about implementing a data extraction usecase. What is the ultimate goal? The ultimate goal need not always be to automate as much as possible: it should be to create value. Because perhaps, you can benefit largely enough from your usecase if Assisted- or Autopilot automation is applied and the extra investment to full automation is not worth it. If we were to take the menu card example from earlier, how would potential time savings look like?

Time savings for automating data extraction (example): 
30 minutes (manual) → 5 minutes (assisted) → 1 minute (autopilot) → 0 minutes (autonomous).

We can see that the time savings are not linear. The more automation we apply, the less time we save. Even though, the last step is the hardest and most expensive to implement. It is not always worth the effort and cost to implement this last step. These are the Diminishing Returns of automation, which can be plotted as follows:

Automation is nice but value is the goal. Take automation step-by-step. Partial automation is also valuable.

Let’s sum things up. To conclude, GenAI for data extraction is:

• Useful for a wide variety of usecases with various data types including text, images, PDFs and audio.
• More likely to provide ROI due to 1) cheaper models, 2) larger context windows and 3) more capable models able to process multi-modal data.
• Very well suited for partial automation: which can bring business value without needing extra investment for full automation.

Good luck with your own GenAI data extraction usecases 🍀♡.

LLM’s can be supercharged using a technique called RAG, allowing us to overcome dealbreaker problems like hallucinations or no access to internal data. RAG is gaining more industry momentum and is becoming rapidly more mature both in the open-source world and at major Cloud vendors. But what can we expect from RAG? What is the current state of the tech in the industry? What use cases work well and which are more challenging? Let’s find out together!

Why RAG

Retrieval Augmented Generation (RAG) is a popular technique to combine retrieval methods like vector search together with Large Language Models (LLM’s). This gives us several advantages like retrieving extra information based on a user search query: allowing us to quote and cite LLM-generated answers. In short, RAG is valuable because:

• Access to up-to-date knowledge
• Access to internal company data
• More factual answers

That all sounds great. But how does RAG work? Let’s introduce the Levels of RAG to help us understand RAG in increasing grades of complexity.

The Levels of RAG

RAG has many different facets. To help more easily understand RAG, let’s break it down into four levels:

The Levels of RAG.

Level 1 Basic RAG	Level 2 Hybrid Search	Level 3 Advanced data formats	Level 4 Multimodal

Each level adds new complexity. Besides explaining the techniques, we will also look into justifying the introduced complexities. That is important, because you want to have good reasons to do so: we want to make everything as simple as possible, but no simpler ^[1].

We will start with building a RAG from the beginning and understand which components are required to do so. So let’s jump right into Level 1: Basic RAG.

Level 1: Basic RAG

Let’s build a RAG. To do RAG, we need two main components: document retrieval and answer generation.

The two main components of RAG: document retrieval and answer generation.

In contrast with a normal LLM interaction, we are now first retrieving relevant context to only then answer the question using that context. That allows us to ground our answer in the retrieved context, making the answer more factually reliable.

Let’s look at both components in more detail, starting with the Document retrieval step. One of the main techniques powering our retrieval step is Vector Search.

Vector Search

To retrieve documents relevant to our user query, we will use Vector Search. This technique is based on vector embeddings. What are those? Imagine we embed words. Then words that are semantically similar should be closer together in the embedding space. We can do the same for sentences, paragraphs, or even whole documents. Such an embedding is typically represented by vectors of 768-, 1024-, or even 3072 dimensions. Though we as humans cannot visualize such high-dimensional spaces: we can only see 3 dimensions! For example sake let us compress such an embedding space into 2 dimensions so we visualize it:

Embeddings that are similar in meaning are closer to each other. In practice not in 2D but higher dimensions like 768D.

Note this is a drastically oversimplified explanation of vector embeddings. Creating vector embeddings of text: from words up to entire documents, is quite a study on its own. Most important to note though is that with embeddings we capture the meaning of the embedded text!

So how to use this for RAG? Well, instead of embedding words we can embed our source documents instead. We can then also embed the user question and then perform a vector similarity search on those:

Embedding both our source documents and the search query allows us to do a vector similarity search.

Great! We now have the ingredients necessary to construct our first Document retrieval setup:

The full retrieval step with vector search.

Next is the Answer generation step. This will entail passing the found pieces of context to an LLM to form a final answer out of it. We will keep that simple so will use a single prompt for that:

The RAG answer generation step. Prompt from LangChain hub (rlm/rag-prompt).

Cool. That concludes our full first version of the Basic RAG. To test our RAG system, we need some data. I recently went to PyData and figured how cool would it be to create a RAG based on their schedule. Let’s design a RAG using the schedule of PyData Eindhoven 2024.

The RAG system loaded with data from the PyData Eindhoven 2024 schedule.

So how do we ingest such a schedule in a vector database? We will take each session and format it as Markdown, respecting the structure of the schedule by using headers.

Each session is formatted as Markdown and then converted to an embedding.

Our RAG system is now fully functional. We embedded all sessions and ingested them into a vector database. We can then find sessions similar to the user question using vector similarity search and answer the question based on a predefined prompt. Let’s test it out!

Testing out our basic RAG system. The RAG retrieves talks related to the user question and correctly answers the question.

That is cool! Our RAG was able to correctly answer the question. Upon inspection of the schedule we can see that the listed talks are indeed all related to sports. We just built a first RAG system 🎉.

There are points of improvement, however. We are now embedding the sessions in its entirety. But embedding large pieces of text can be problematic, because:

• ❌ Embeddings can get saturated and lose meaning
• ❌ Imprecise citations
• ❌ Large context → high cost

So what we can do to solve this problem is to divide up text in smaller pieces, then embed those instead. This is Chunking.

Chunking

In chunking the challenge lies in determining how to divide up the text to then embed those smaller pieces. There are many ways to chunk text. Let’s first look at a simple one. We will create chunks of fixed length:

Splitting a text into fixed-sized pieces does not create meaningful chunks. Words and sentences are split in random places. 
Character Text Splitter with chunk_size = 25. This is an unrealistically small chunk size but is merely used for example sake.

This is not ideal. Words, sentences and paragraphs are not respected and are split at inconvenient locations. This decreases the quality of our embeddings. We can do better than this. Let’s try another splitter that tries to better respect the structure of the text by taking into account line breaks (¶):

Text is split, respecting line breaks (¶). This way, chunks do not span across paragraphs and split better. 
Recursive Character Text Splitter with chunk_size = 25. This is an unrealistically small chunk size but is merely used for example sake.

This is better. The quality of our embeddings is now better due to better splits. Note that chunk_size = 25 is only used for example sake. In practice we will use larger chunk sizes like 100, 500, or 1000. Try and see what works best on your data. But most notably, also do experiment with different text splitters. The LangChain Text Splitters section has many available and the internet is full of others.

Now we have chunked our text, we can embed those chunks and ingest them in our vector database. Then, when we let the LLM answer our question we can also ask it to state which chunks it used in answering the question. This way, we can pinpoint accurately which information the LLM was grounded in whilst answering the question, allowing us to provide the user with Citations:

Chunks can be used to show user where the information came from to generate the final answer, allowing us to show the user source citations.

That is great. Citations can be very powerful in a RAG application to improve transparency and thereby also user trust. Summarized, chunking has the following benefits:

• Embeddings are more meaningful
• Precise citations
• Shorter context → lower cost

We can now extend our Document retrieval step with chunking:

RAG retrieval step with chunking.

We have our Basic RAG set up with Vector search and Chunking. We also saw our system can answer questions correctly. But how well actually does it do? Let’s take the question “What’s the talk starting with Maximizing about?” and launch it at our RAG:

For this question the incorrect context is retrieved, causing the LLM to give a wrong answer.
UI shown is LangSmith, a GenAI monitoring tool. Open source alternative is LangFuse.

Ouch! This answer is wildly wrong. This is not the talk starting with Maximizing. The talk described has the title Scikit-Learn can do THAT?!, which clearly does not start with the word Maximizing.

For this reason, we need another method of search, like keyword search. Because we would also still like to keep the benefits of vector search, we can combine the two methods to create a Hybrid Search.

Level 2: Hybrid Search

In Hybrid Search, we aim to combine two ranking methods to get the best of both worlds. We will combine Vector search with Keyword search to create a Hybrid Search setup. To do so, we must pick a suitable ranking method. Common ranking algorithms are:

• TF-IDF: Term Frequency-Inverse Document Frequency
• Okapi BM-25: Okapi Best Matching 25

… of which we can consider BM-25 an improved version of TF-IDF. Now, how do we combine Vector Search with an algorithm like BM-25? We now have two separate rankings we want to fuse together into one. For this, we can use Reciprocal Rank Fusion:

Reciprocal Rank Fusion is used to merge two ranking methods into one (see Elastic Search docs).

Reciprocal Rank Fusion is massively useful to combine two rankings. This way, we can now use both Vector search and Keyword search to retrieve documents. We can now extend again our Document retrieval step to create a Hybrid Search setup:

The RAG retrieval step with hybrid search.

Most notably, when a user performs a search, a query is made both to our vector database and to our keyword search. Once the results are merged using Reciprocal Rank Fusion, the top results are taken and passed back to our LLM.

Let’s take again the question “What’s the talk starting with Maximizing about?” like we did in Level 1 and see how our RAG handles it with Hybrid Search:

Hybrid Search allows us to combine Vector search and Keyword search, allowing us to retrieve the desired document containing the keyword.

That is much better! 👏🏻 The document we were looking for was now ranked high enough that it shows up in our retrieval step. It did so by boosting the terms used in the search query. Without the document available in our prompt context, the LLM could impossibly give us the correct answer. Let’s see both the retrieval step and generation step to see what our LLM now answers on this question:

With the correct context, we also got the correct answer!

That is the correct answer ✓. With the right context available to the LLM we also get the right answer. Retrieving the right context is the most important feature of our RAG: without it, the LLM can impossibly give us the right answer.

We now learned how to build a Basic RAG system and how to improve it with Hybrid Search. The data we loaded in comes from the PyData Eindhoven 2024 schedule: which was actually conveniently available in JSON format. But what about other data formats? In the real world, we can be asked to ingest other formats into our RAG like HTML, Word, and PDF.

Having structured data available like JSON is great, but that is not always the case in real life…

Formats like HTML, Word and especially PDF can be very unpredictable in terms of structure, making it hard for us to parse them consistently. PDF documents can contain images, graphs, tables, text, basically anything. So let us take on this challenge and level up to Level 3: Advanced data formats.

Level 3: Advanced data formats

This level revolves around ingesting challenging data formats like HTML, Word or PDF into our RAG. This requires extra considerations to do properly. For now, we will focus on PDF’s.

So, let us take some PDF’s as an example. I found some documents related to Space X’s Falcon 9 rocket:

Falcon 9 related documents we are going to build a RAG with.

User’s guide (pdf)	Cost estimates (pdf)	Capabilities & Services (pdf)

We will now want to first parse those documents into raw text, such that we can then chunk- and embed that text. To do so, we will use a PDF parser for Python like pypdf. Conveniently, there’s a LangChain loader available for pypdf:

PDF to text parsing using PyPDFLoader.

Using pypdf we could convert these PDF’s into raw text. Note there’s many more available, check out the LangChain API reference. Both offline, local and Cloud solutions are offered like GCP Document AI, Azure Document Intelligence or Amazon Textract.

With such a document parsing step set up, we will need to extend our Document retrieval step to cover this new component:

RAG retrieval with a document parsing step, like converting PDF to raw text.

Time to test out our RAG! We embedded and ingested the raw text into our vector database and can now make queries against it. Let’s ask about the cost of a Falcon 9 rocket launch:

Our RAG can now answer questions about the PDF’s.

Awesome, that is the correct answer. Let’s try another question: “What is the payload I can carry with the Falcon 9 rocket to Mars?“:

Ouch! Our RAG got the answer wrong. It suggests that we can bring 8,300 kg of payload to Mars with a Falcon 9 rocket whilst in reality, this is 4,020 kg. That’s no small error.

Ow! Our RAG got that answer all wrong. It suggests we can bring double the payload to Mars than what is allowed. That is pretty inconvenient, in case you were preparing for a trip to Mars 😉.

We need to debug what went wrong. Let’s look at the context that was passed to the LLM:

The context provided to the LLM contains jumbled and unformatted text originating from a table. It should therefore come as no surprise that our LLM has difficulties answering questions about this table.

That explains a bunch. The context we’re passing to the LLM is hard to read and has a table encoded in a jumbled way. Like us the LLM has a hard time making sense of this. Therefore, we need to better encode this information in the prompt so our LLM can understand it.

If we want to support tables we can introduce an extra processing step. One option is to use Computer vision models to detect tables inside our documents, like table-transformer. If a table gets detected we can then give it a special treatment. What we can for example do, is encode tables in our prompt as Markdown:

First parsing our table into a native format and then converting it to Markdown allows our LLM to much better understand the table.

Having detected the table and parsed it into a native format in our Python code allows us to then encode it in Markdown. Let’s pass that to our LLM instead and see what it answers this time:

Our RAG got the answer correct this time.

Hurray! We got it correct this time. The LLM we used could easily interpret the Markdown table and pinpoint the correct value to use in answering the question. Note that still, we need to be able to retrieve the table in our retrieval step. This setup assumes we built a retrieval step that is able to retrieve the table given the user question.

However, I have to admit something. The model we have been using for this task was GPT-3.5 turbo, which is a text-only model. Newer models have been released that can handle more than just text, which are Multimodal models. After all, we are dealing with PDF’s which can also be seen as a series of images. Can we leverage such multimodal models to better answer our questions? Let’s find out in Level 4: Multimodal.

Level 4: Multimodal

In this final level, we will look into leveraging the possibilities of Multimodal models. One of them is GPT-4o, which was announced May 2024:

GPT-4o is a multimodal model that can reason across audio, vision and text. Launched by OpenAI in May 2024.

This is a very powerful model that can understand audio, vision and text. This means we can feed it images as part of an input prompt. Given that we can in our retrieval step retrieve the right PDF pages to use, we can insert those images in the prompt and ask the LLM our original question. This has the advantage that we can understand content that was previously very challenging to encode in just text. Also, content we interpret and encode as text is exposed to more conversion steps exposing us to risk of information getting lost in translation.

For example sake we will take the same table we had before but then answer the question using a Multimodal model. We can take the retrieved PDF pages encoded as images and insert them right into the prompt:

With a Multimodal model, we can insert an image in the prompt and let the LLM answer questions about it.

Impressive. The LLM got the answer correct. We should be aware though, that inserting images in the prompt comes with a very different token usage than the Markdown table we inserted as text before:

When using Multimodal models, do be aware of the extra cost that comes with it.

That is an immense increase in cost. Multimodal models can be incredibly powerful to interpret content that is otherwise very difficult to encode in text, as long as it is worth the cost ✓.

Concluding

We have explored RAG in 4 levels of complexity. We went from building our first basic RAG to a RAG that leverages Multimodal models to answer questions based on complex documents. Each level introduces new complexities which are justified in each their own way. Summarising, the Levels of RAG are:

The Levels of RAG, summarised.

Level 1 Basic RAG	Level 2 Hybrid Search	Level 3 Advanced data formats	Level 4 Multimodal
RAG’s main steps are 1) retrieval and 2) generation. Important components to do so are Embedding, Vector Search using a Vector database, Chunking and a Large Language Model (LLM).	Combining vector search and keyword search can improve retrieval performance. Sparse text search can be done using: TF-IDF and BM- 25. Reciprocal Rank Fusion can be used to merge two search engine rankings.	Support formats like HTML, Word and PDF. PDF can contain images, graphs but also tables. Tables need a separate treatment, for example with Computer Vision, to then expose the table to the LLM as Markdown.	Multimodal models can reason across audio, images and even video. Such models can help process complex data formats, for example by exposing PDF’s as images to the model. Given that the extra cost is worth the benefit, such models can be incredibly powerful.

RAG is a very powerful technique which can open up many new possibilities at companies. The Levels of RAG help you reason about the complexity of your RAG and allow you to understand what is difficult to do with RAG and what is easier. So: what is your level? 🫵

We wish you all the best with building your own RAG 👏🏻.

Talk held together with Sander van Donkelaar.

AI Community Day - Utrecht

AI Community Day - Utrecht

Global AI Community

Google Cloud Platform has an increasing set of managed services that can help you build production-ready Retrieval-Augmented Generation applications. Services like Vertex AI Search & Conversation and Vertex AI Vector Search give us scalability and ease of use. How can you best leverage them to build RAG applications? Let’s explore together. Read along!

Retrieval Augmented Generation

Even though Retrieval-Augmented Generation (RAG) was coined already in 2020, the technique got supercharged since the rise of Large Language Models (LLMs). With RAG, LLMs are combined with search techniques like vector search to enable realtime and efficient lookup of information that is outside the model’s knowledge. This opens up many exciting new possibilities. Whereas previously interactions with LLMs were limited to the model’s knowledge, with RAG it is now possible to load in company internal data like knowledge bases. Additionally, by instructing the LLM to always ‘ground’ its answer based on factual data, hallucinations can be reduced.

Why RAG?

Let’s first take a step back. How exactly can RAG benefit us? When we interact with an LLM, all its factual knowledge is stored inside the model weights. The model weights are set during its training phase, which can be a while ago. In fact, this can be more than a year.

Additionally, these publically offered models are trained on mostly public data. If you want to use company internal data, an option is to fine-tune or retrain the model, which can be expensive and time-consuming.

This boils down to three main limitations: the models knowledge is outdated, the model has no access to internal data, and the model can hallucinate answers.

With RAG we can circumvent these limitations. Given the question a user has, information relevant to that question can be retrieved first to then be presented to the LLM.

The LLM can then augment its answer with the retrieved information to generate a factual, up-to-date and yet human-readable answer. The LLM is to be instructed to at all times ground its answer in the retrieved information, which can help reduce hallucinations.

These benefits are great. So how do we actually build a RAG system?

Building a RAG system

In a RAG system, there are two main steps: 1) Document retrieval and 2) Answer generation. Whereas the document retrieval is responsible for finding the most relevant information given the user’s question, the answer generation is responsible for generating a human-readable answer based on information found in the retrieval step. Let’s take a look at both in more detail.

Document retrieval

First, Document retrieval. Documents are converted to plain text and chunked. The chunks are then embedded and stored in a vector database. User questions are also embedded, enabling a vector similarity search to obtain the best matching documents. Optionally, a step can be added to extract document metadata like title, author, summary, keywords, etc, which can subsequently be used to perform a keyword search. This can all be illustrated like so:

Neat. But what about GCP. We can map the former to GCP as follows:

Document AI is used to process documents and extract text, Gemini and textembedding-gecko to generate metadata and embeddings respectively and Vertex AI Vector Search is used to store the embeddings and perform similarity search. By using these services, we can build a scalable retrieval step.

Answer generation

Then, Answer generation. We will need an LLM for this and instruct it to use the provided documents. We can illustrate this like so:

Here, the documents can be formatted using an arbitrary function that generates valid markdown.

We have already come across multiple GCP services that can help us build a RAG system. So now, what other offerings does GCP have to help us build a RAG system and what flavours are there to combine services?

The RAG flavours on GCP

So far, we have seen GCP services that can help us build a RAG system. These include Document AI, Vertex AI Vector Search, Gemini Pro, Cloud Storage and Cloud Run. But GCP also has Vertex AI Search & Conversation.

Vertex AI Search & Conversation is a service tailored to GenAI usecases, built to do some of the heavy lifting for us. It can ingest documents, create embeddings and manage the vector database. You just have to focus on ingesting data in the correct format. Then, you can use Search & Conversation in multiple ways. You can either get only search results, given a search query, or you can let Search & Conversation generate a full answer for you with source citations.

Even though Vertex AI Search & Conversation is very powerful, there can be scenarios when you want more control. Let’s take a look on these levels of going either managed or remaining in-control.

The easiest way to get started with RAG on GCP is to use Search & Conversation. The service can ingest documents from multiple sources like Big Query and Google Cloud Storage. Once those are ingested, it can generate answers backed by citations for you. This is simply illustrated like so:

If you want more control, you can use Gemini for answer generation instead of letting Search & Conversation do it for us. This way, you can have more control to do any prompt engineering you like.

Lastly, you can have full control over the RAG system. This means you have to manage both the document retrieval and answer generation yourself. This does mean more manual work in engineering the system. Documents can be processed by Document AI, chunked, embedded and its vectors stored in Vertex AI Vector Search. Then Gemini can be used to generate the final answers.

The advantage here is that you are able to fully control yourself how you process the documents an convert them to embeddings. You can use all of Document AI’s Processors offering to process the documents in different ways.

Do take the the managed versus manual approach tradeoffs into consideration. Ask yourself questions like:

• How much time and energy do you want to invest building something custom for the flexibility that you need?
• Do you really need the flexibility for your solution to give in to extra maintenance cost?
• Do you have the engineering capacity to build- and maintain a custom solution?
• Are the initially invested build costs worth the money saved in not using a managed solution?

So then, you can decide what works best for you ✓.

Concluding

RAG is a powerful way to augment LLMs with external data. This can help reduce hallucinations and provide more factual answers. At the core of RAG systems are document processors, vector databases and LLM’s.

Google Cloud Platform offers services that can help build production-ready RAG solutions. We have described three levels of control in deploying a RAG application on GCP:

• Fully managed: using Search & Conversation.
• Partly managed: managed search using Search & Conversation but manual prompt-engineering using Gemini.
• Full control: manual document processing using Document AI, embedding creation and vector database management using Vertex AI Vector Search.

That said, I wish you good luck implementing your own RAG system. Use RAG for great good! ♡