Building Multimodal AI Applications with Gemma 4 and Transformers

In the first part of this series, we explored the architecture behind Gemma 4: the interleaved attention design, Mixture-of-Experts routing, multimodal encoders, Per-Layer Embeddings, and the engineering decisions that allow the model family to scale from smartphones to large GPU servers. We also looked at the different variants, hardware requirements, benchmarks, and the reasoning capabilities that make Gemma 4 one of the most compelling open-weight model releases today.

But architecture alone does not tell the full story.

The real question for most developers is simple: how do we actually use Gemma 4 in practice?

That is what this second part focuses on. Instead of discussing theory, we will build directly with the model using Hugging Face Transformers. We will see how Gemma 4 handles images, videos, audio, structured tool calling, object detection, screenshot-to-code generation, and multimodal reasoning, all from a single unified interface.

One of the most interesting things about Gemma 4 is that the workflow stays surprisingly consistent across tasks. Whether we are passing a webpage screenshot, an audio clip, or a video file, the overall pipeline barely changes:

• Define the multimodal message.
• Process it using the Gemma processor.
• Generate outputs from the model.
• Decode and parse the response.

Once this pattern becomes familiar, building multimodal applications starts feeling much more approachable.

In this lesson, we will go section by section through the notebook and explain not just what the code is doing, but why each step matters. Along the way, we will also connect the implementation back to the architectural ideas from Part 1, showing how concepts (e.g., multimodal encoders, thinking mode, and structured outputs) appear in real inference pipelines.

This lesson is the 2nd in a 5-part series on Google DeepMind’s Gemma 4:

1. Google DeepMind’s Gemma 4: MoE, Efficiency Tricks, and Benchmarks
2. Building Multimodal AI Applications with Gemma 4 and Transformers (this tutorial)
3. Lesson 3
4. Lesson 4
5. Lesson 5

To learn how to build multimodal applications with Gemma 4, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Configuring Your Development Environment

To follow this guide, you need to have the following libraries installed on your system.

!pip install -q -U transformers

This command installs the latest version of the transformers library.

Here:

• -U upgrades the package to the newest available version
• -q enables quiet mode to reduce unnecessary installation logs

Using the latest version is important because Gemma 4 support, multimodal processors, and newer generation features are included in recent releases of the library.

The transformers package gives us access to:

• Pretrained Gemma 4 checkpoints
• Multimodal processors
• Tokenization utilities
• Chat templates
• Text generation APIs

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Installing Python Dependencies and Importing Gemma 4 Multimodal Libraries

After configuring the environment, the next step is importing all the required libraries. These imports provide everything needed for loading Gemma 4, processing multimodal inputs, generating outputs, and visualizing results.

import re
import json
import torch
import matplotlib.pyplot as plt
import matplotlib.patches as patches

from PIL import Image
from transformers.image_utils import load_image
from transformers import AutoModelForMultimodalLM, AutoProcessor

We first import Python’s built-in re and json libraries. The re library is used for regular expression operations. Later in the lesson, we use it while parsing structured outputs such as bounding box predictions returned by the model.

The json library helps us work with JSON responses generated by Gemma 4. This becomes especially useful for tasks (e.g., object detection and function calling), where the model produces structured outputs instead of plain text.

Next, we import PyTorch. PyTorch is the deep learning framework powering the model inference pipeline. Hugging Face Transformers internally relies on PyTorch tensors for:

• Token representations
• Model weights
• GPU computation
• Generation operations

Almost every operation inside Gemma 4 inference eventually runs through PyTorch.

We then import Matplotlib utilities. These libraries are used for visualization. In the object detection section later in the lesson, we will draw predicted bounding boxes on images using rectangle patches.

Next, we import PIL. PIL (Python Imaging Library) is widely used for image loading and manipulation in Python applications. It helps us work with image files before sending them into the model.

We also import load_image. This utility simplifies image loading directly from URLs or local paths. Instead of manually downloading and preprocessing images, we can load them with a single function call.

Finally, we import the 2 most important components from Transformers.

AutoModelForMultimodalLM loads the Gemma 4 multimodal model itself. Since Gemma 4 can process text, images, video, and audio, we use the multimodal model interface instead of a text-only causal language model.

AutoProcessor handles preprocessing for all supported modalities. It prepares:

• Text tokens
• Image embeddings
• Audio features
• Video inputs
• Chat templates

into the format expected by Gemma 4.

Together, these imports form the foundation for the rest of the lesson. Once these libraries are loaded, we are ready to initialize the model and begin multimodal inference.

Loading the Gemma 4 Multimodal Model with Hugging Face Transformers

Now that the environment and dependencies are ready, we can load the Gemma 4 model and its processor.

model_id = "google/gemma-4-E2B-it"
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

We first define the model checkpoint: google/gemma-4-E2B-it.

Here, we are using the instruction-tuned Gemma-4-E2B-it variant. The it suffix stands for instruction tuned, meaning the model has been optimized for conversational and task-following behavior.

The Gemma 4 E2B model is also one of the smaller Gemma 4 variants, making it practical for experimentation and multimodal inference without requiring extremely large GPU memory. As discussed in the first part of this series, the E-series models are specifically designed for efficient deployment while still supporting text, image, video, and audio understanding.

Next, we load the model itself. The from_pretrained() method downloads the pretrained weights directly from Hugging Face and initializes the model architecture automatically.

The important argument here is: device_map="auto". This tells Transformers to automatically decide where the model should be loaded:

• GPU if CUDA is available
• CPU otherwise

For larger models, this can also distribute layers across multiple GPUs automatically. In our case, it simplifies deployment because we do not need to manually move tensors or model weights between devices.

Finally, we load the processor. The processor is responsible for preparing multimodal inputs before they are passed into Gemma 4.

This includes:

• Tokenizing text
• Processing images
• Handling audio inputs
• Formatting video data
• Applying the correct chat template

One of the biggest advantages of the processor API is consistency. Whether we send text, images, video, or audio into the model, the processor converts everything into the exact format expected by Gemma 4 internally.

At this point, both the model and processor are fully initialized, and we are ready to begin multimodal inference tasks.

Screenshot-to-Code Generation with Gemma 4 Vision-Language AI

One of the most impressive capabilities of Gemma 4 is multimodal code generation. Instead of giving the model only text prompts, we can provide an image of a webpage and ask it to generate the corresponding HTML code.

We begin by defining the multimodal conversation input:

messages = [
   {
       "role": "user",
       "content": [
           {
               "type": "image",
               "image": "https://github.com/PyImageSearch/assets/blob/main/images/landing-page-1.png?raw=true",
           },
           {"type": "text", "text": "Write HTML code for this page."},
       ],
   }
]

Here, the input follows a chat-style structure. Each message contains:

• A role
• A content field

The content itself is a list because Gemma 4 supports multiple modalities within the same conversation turn.

The first content item is the webpage screenshot. This tells the processor that the input modality is an image. Instead of manually downloading and preprocessing the image ourselves, we simply provide the URL.

The second content item is the text instruction. This prompt guides the model toward the task we want it to perform.

Together, the image and text become a unified multimodal input. Gemma 4 processes both inputs simultaneously, allowing it to reason about the webpage layout visually while generating HTML code as output.

inputs = processor.apply_chat_template(
   messages,
   tokenize=True,
   return_dict=True,
   return_tensors="pt",
   add_generation_prompt=True,
   enable_thinking=True,
).to(model.device)

Next, we prepare the inputs using the processor. This is one of the most important steps in the entire workflow.

The apply_chat_template() function converts the multimodal conversation into the exact format expected by Gemma 4 internally.

Let us break down the major arguments.

• tokenize=True: converts the processed input into tokens that the model can understand
• return_tensors="pt": tells the processor to return PyTorch tensors instead of Python lists
• return_dict=True: returns a structured dictionary containing all required tensors instead of returning only token IDs
• add_generation_prompt=True: appends the assistant generation marker so the model knows it should begin generating a response
• enable_thinking=True: activates Gemma 4’s thinking mode. As discussed in Part 1, thinking mode encourages the model to internally reason through the problem before producing the final answer.

Finally, we move all tensors onto the same device as the model. This ensures inference runs correctly on either CPU or GPU.

output = model.generate(**inputs, max_new_tokens=4000)

Once the inputs are ready, we generate the output. The generate() function performs autoregressive decoding and produces the model response token by token. max_new_tokens=4000 sets the maximum generation length. Since HTML generation can produce large outputs, we allow a relatively high token budget.

input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)

Next, we extract only the generated portion of the response. This step is necessary because the generated output contains:

• The original input tokens
• The newly generated tokens

We remove the input portion and keep only the generated response. We then decode the generated token IDs back into readable text. The skip_special_tokens=True argument removes internal formatting tokens that are not useful for display.

result = processor.parse_response(generated_text)

print(result["content"])

Next, we parse the model response. This converts the raw generated text into a cleaner structured format. Finally, we print the generated HTML output.

Output

At this stage, Gemma 4 produces HTML code that attempts to recreate the webpage shown in the screenshot.

This example demonstrates one of the most exciting aspects of multimodal language models: the ability to translate visual layouts directly into working code. Instead of treating images and programming as separate domains, Gemma 4 combines visual understanding and code generation into a single inference pipeline.

thought
Here's a thinking process that leads to the suggested HTML code:
1.  **Analyze the Request:** The user wants HTML code for the provided image (a website screenshot/design).
2.  **Deconstruct the Image (Identify Key Components & Structure):**
    *   **Header/Navigation Bar:**
        *   Logo: "YOURLOGO" (Top left).
        *   Navigation Links: "Home," "Company," "Services," "Contact."
        *   Icon: Search/Magnifying glass (Top right).
    *   **Hero Section/Main Content Area:**
        *   *Visual Element (Left/Center):* A stylized illustration of a woman celebrating, surrounded by stars, planets, and dots.
        *   *Text/Headline (Right):*
            *   Headline: "Creative Ideas" (Large, orange/bold).
            *   Subheadline/Tagline: "A Studio based in New York, US."
            *   Body Text: "Solutions of design in modern era, consectetur adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat." (Placeholder text).
        *   *Call to Action (CTA):* A button labeled "SUBSCRIBE" (Dark/Maroon).
        *   *Social Media Icons:* Facebook, Instagram, Twitter (Small icons below the CTA).
...
...

We can see that Gemma 4 not only understands the webpage visually, but also reasons through its structure before generating the final code. The model identifies components such as the navigation bar, hero section, buttons, and social icons, and then converts them into structured HTML and CSS.

Interestingly, the output also includes responsive styling, layout organization, practical implementation notes, and even a dedicated section explaining how to use the generated code. We followed the same instructions provided by Gemma 4 to run the generated HTML and CSS files, and the resulting webpage closely matched the original design shown in the input image.

Video Understanding and Multimodal Reasoning with Gemma 4

After testing image understanding and code generation, we can move one step further and evaluate Gemma 4 on video reasoning tasks.

messages = [
   {
       "role": "user",
       "content": [
           {"type": "video", "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"},
           {"type": "text", "text": "What is happening in the video? What is the song about?"},
       ],
   },
]

Here, instead of providing an image, we provide a video file as input.

The first content block defines the video source. This tells the processor that the input modality is a video rather than an image or audio clip.

The second content block contains the text instruction. This prompt asks the model to reason about:

• The visual scene
• The event taking place
• The semantic meaning of the song

This is important because the model is not simply captioning individual frames. It is performing multimodal reasoning across both temporal and audio information.

inputs = processor.apply_chat_template(
   messages,
   tokenize=True,
   return_dict=True,
   return_tensors="pt",
   add_generation_prompt=True,
   load_audio_from_video=True,
).to(model.device)

Next, we process the multimodal input. Most of the arguments are similar to the previous example, but there is one important addition here:

• load_audio_from_video=True flag tells the processor to extract the audio stream from the video alongside the visual frames.

As discussed in the first part of this series, the smaller Gemma 4 E-series models support audio understanding in addition to image and text reasoning.

This means the model can jointly analyze:

• Video frames
• Speech
• Music
• Ambient sounds

instead of relying only on visual information.

In practice, this allows Gemma 4 to answer richer questions about videos, such as:

• Understanding spoken dialogue
• Identifying music genres
• Inferring emotional tone
• Explaining events happening in the scene

output = model.generate(**inputs, max_new_tokens=200)

Once the inputs are prepared, we generate the output. Here, max_new_tokens=200 limits the response length. Since video descriptions are usually shorter than HTML generation tasks, a smaller token budget is sufficient.

input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
result = processor.parse_response(generated_text)

After generation, we extract only the generated portion of the sequence. Just as before, this removes the original input tokens and keeps only the newly generated response.

Next, we decode the generated tokens back into readable text. We then parse the response into a cleaner structure.

print(result["content"])

Finally, we print the generated output.

Output

At this stage, Gemma 4 analyzes the concert video and produces a multimodal interpretation of the scene.

This example highlights an important capability of Gemma 4: unified video understanding. Instead of requiring separate models for:

• Video captioning
• Audio transcription
• Scene understanding
• Semantic reasoning

Gemma 4 performs these tasks inside a single multimodal inference pipeline.

Based on the video, here is what is happening:
**What is happening in the video?**
The video captures a live concert performance on a large stage. We see a band performing, with musicians playing instruments (guitars, drums, etc.) under bright stage lights and dramatic blue and white lighting, including significant smoke/fog effects. In the foreground, there is a crowd of people watching the show, with several audience members visible from behind. The energy of the performance seems high, as suggested by the lighting and the engagement of the audience.
**What is the song about?**
The provided lyrics are:
> "I'm falling on the street
> From neck to chest
> Could it be that moments
> Another one I want to be is
> In the storm alone
> I've been all alone
> I never want to see
> Oh this nice sad place you was given fate
> I don't want it

Multimodal Function Calling with Gemma 4

One of the most powerful capabilities introduced in modern language models is function calling. Instead of generating only plain text responses, the model can decide when to invoke external tools and generate structured arguments for them.

In this example, we combine:

• Vision understanding
• Reasoning
• Tool usage

inside a single multimodal workflow.

WEATHER_TOOL = {
   "type": "function",
   "function": {
       "name": "get_weather",
       "description": "Gets the current weather for a specific location.",
       "parameters": {
           "type": "object",
           "properties": {
               "city": {"type": "string", "description": "The city name"},
           },
           "required": ["city"],
       },
   },
}
tools = [WEATHER_TOOL]

We first define a weather tool schema. This structure describes the external tool available to the model. The tool definition contains several important components.

First, we specify the tool type: "type": "function" tells Gemma 4 that the tool represents a callable function.

Next, we define the actual function metadata. "name": "get_weather" is the function name the model will reference when generating tool calls. We also provide a natural language description. This helps the model understand what the tool does and when it should be used.

Next comes the parameter schema. This follows a JSON-style schema format. Here, the function expects a single required argument:

• city

The model will later generate this parameter automatically based on the image and user request. We then place the tool into a list. This allows multiple tools to be passed into the model if needed.

messages = [
   {"role": "user", "content": [
       {"type": "image", "image": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/venice.jpg"},
       {"type": "text", "text": "What is the city in this image? Check the weather there right now."},
   ]},
]

Next, we define the multimodal user input.

This input combines:

• An image
• A reasoning instruction

The image contains a photograph of Venice, while the text prompt asks the model to:

• Identify the city
• Use the weather tool for that location

This is important because the model must first perform visual reasoning before it can invoke the function correctly.

inputs = processor.apply_chat_template(
   messages,
   tools=[WEATHER_TOOL],
   tokenize=True,
   return_dict=True,
   return_tensors="pt",
   add_generation_prompt=True,
   enable_thinking=True,
).to(model.device)

Next, we prepare the inputs.

The major addition here is: tools=[WEATHER_TOOL]

This injects the available tool schema directly into the prompt context. Gemma 4 now knows:

• Which tools exist
• What each tool does
• Which parameters are required

We also enable reasoning mode again: enable_thinking=True

This helps the model reason through the sequence:

• Analyze image
• Infer city
• Decide tool usage
• Generate function arguments

output = model.generate(**inputs, max_new_tokens=1000)

After processing the inputs, we generate the output. Since function-calling outputs can include reasoning traces and structured responses, we allow a larger token budget.

input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
result = processor.parse_response(generated_text)

Next, we extract only the generated tokens. We then decode the generated response. After decoding, we parse the structured response.

print(result["content"])

Finally, we print the generated output.

Output

At this stage, Gemma 4 identifies the city from the image and generates a structured tool call for retrieving the weather.

This is the input image:

This is the generated output:

thought
Here's a thinking process to arrive at the desired response:
1.  **Analyze the Request:** The user has provided an image and asked two questions:
    *   "What is the city in this image?"
    *   "Check the weather there right now."
2.  **Analyze the Image:**
    *   The image shows a large, ornate building situated on the water, with boats and other buildings in the background.
    *   The architecture strongly resembles famous Italian landmarks, specifically those found in Venice (bridges, canals, specific facade styles).
    *   The presence of a canal setting is a huge clue.
3.  **Identify the Location (Inference/Knowledge Retrieval):**
    *   The architecture points strongly toward Venice, Italy.
    *   The specific building looks like a prominent structure in Venice (e.g., a church or palace facade).
4.  **Determine the Necessary Tool Call:**
    *   The user wants the weather for the identified city.
    *   The available tool is `get_weather(city: STRING)`.
5.  **Formulate the Plan:**
    *   **Step 1:** State the identified city.
    *   **Step 2:** Call the `get_weather` tool with the identified city.

6.  **Execute Step 1 (Internal Check):** The city is Venice.
7.  **Execute Step 2 (Tool Call Generation):**
    *   Call: `get_weather(city="Venice")`
8.  **Final Output Generation:** Since I need to call a tool, I must output the tool call first. (Self-Correction: I cannot give the weather *yet*; I must request the tool execution.)call:get_weather{city:Venice}

This example demonstrates why multimodal function calling is so powerful. The model is no longer limited to passive question answering. Instead, it can:

• Observe visual information
• Reason about the task
• Select external tools
• Generate structured arguments

This forms the foundation for agentic AI systems where multimodal models interact with application programming interfaces (APIs), databases, and external applications dynamically.

Object Detection and Visual Grounding with Gemma 4

Beyond captioning and reasoning, Gemma 4 can also perform structured visual localization tasks such as object detection and pointing. Instead of generating only natural language descriptions, the model can return bounding box coordinates for objects inside an image.

image_url = "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/bird.png"
image = load_image(image_url)

We begin by loading the image. Here, load_image() downloads the image directly from the URL and converts it into a format suitable for further processing.

def resize_to_48_multiple(image):
   w, h = image.size
   new_w = (w // 48) * 48
   new_h = (h // 48) * 48
   return image.crop((0, 0, new_w, new_h))

Next, we define a preprocessing helper function. This function adjusts the image dimensions so both height and width become multiples of 48.

We first extract the original image dimensions. Next, we compute the nearest lower multiples of 48. The // operator performs integer division. This effectively rounds the dimensions downward to the nearest valid multiple.

Finally, we crop the image. This preprocessing step helps ensure compatibility with the vision encoder and patch processing pipeline used internally by Gemma 4. Since transformer-based vision models often process images in fixed patch sizes, maintaining aligned dimensions simplifies inference.

def inputs_for_object_detection(image, what_object):
   messages = [
       {
           "role": "user", "content": [
               {"type": "image", "image": image},
               {"type": "text", "text": f"What's the bounding box for the {what_object} in the image?"}
           ]
       }
   ]

   inputs = processor.apply_chat_template(
       messages,
       tokenize=True,
       add_generation_prompt=True,
       return_dict=True,
       return_tensors="pt",
       enable_thinking=False,
   )

   return inputs.to(model.device)

Next, we define the function that prepares the object detection prompt. This function takes:

• The input image
• The target object name

and constructs the multimodal request.

Inside the function, we first define the chat-style message. The image is passed directly as part of the multimodal input.

The text prompt dynamically inserts the target object name using an f-string. For example, if:

what_object = "bird"

the final prompt becomes:

"What's the bounding box for the bird in the image?"

This allows the same pipeline to work for many different object categories without changing the core logic.

Next, we process the multimodal request. Most of the arguments are familiar from earlier examples, but one important detail here is: enable_thinking=False

Unlike reasoning-heavy tasks (e.g., code generation or function calling), object detection is primarily a localization task. Disabling thinking mode helps keep the output concise and focused on structured bounding box predictions instead of extended reasoning traces.

The processor then:

• Tokenizes the text prompt
• Encodes the image
• Applies the correct chat template
• Converts everything into PyTorch tensors

Finally, we move the tensors onto the same device as the model.

def extract_json(text: str):
   text = text.strip()

   text = re.sub(r"^```(?:json)?\s*", "", text)
   text = re.sub(r"\s*```$", "", text)

   # Try direct parse first
   try:
       return json.loads(text)
   except json.JSONDecodeError:
       pass

   # Fallback: extract first JSON object or array
   match = re.search(r'(\{.*\}|\[.*\])', text, re.DOTALL)
   if match:
       candidate = match.group(1)
       return json.loads(candidate)

   raise ValueError("No valid JSON found")

After preparing the object detection inputs, the next step is running inference, extracting the structured bounding box predictions, and visualizing the detected object on the image.

We first define a helper function for parsing the model response.

The generated output from Gemma 4 is usually returned as text, even when it contains structured JSON. This function cleans and extracts the JSON portion safely.

We first remove unnecessary whitespace. Next, we remove markdown-style code fences if they exist.

Sometimes models return outputs such as:

[
 {
   "box_2d": [...]
 }
]

These regular expressions remove the surrounding Markdown formatting so the remaining content becomes valid JSON.

Next, we attempt direct JSON parsing. If the response is already valid JSON, this step succeeds immediately.

However, model outputs are not always perfectly formatted. Sometimes, additional explanation text appears before or after the JSON block. To handle that, we add a fallback mechanism. This searches for the first JSON object {} or JSON array [] inside the generated response.

If a match is found: we extract and parse only the JSON portion.

Finally, if no valid JSON is detected, the function raises an error.

This helper function is important because structured outputs from multimodal models can sometimes contain extra formatting or reasoning traces. Robust parsing makes downstream processing much more reliable.

def detect_object(image_url, what_object):
   image = load_image(image_url)
   image = resize_to_48_multiple(image)
   inputs = inputs_for_object_detection(image, what_object)
   input_len = inputs["input_ids"].shape[-1]
   generated_outputs = model.generate(**inputs, max_new_tokens=1000, do_sample=False)
   generated = processor.decode(generated_outputs[0, input_len:])
   parsed_json = extract_json(generated)[0]
   return parsed_json

Next, we define the main object detection function. This function performs the complete detection pipeline.

We first load and preprocess the image. Next, we prepare the detection inputs. We then compute the input sequence length. This helps us later separate the generated response from the original prompt tokens.

Next comes inference. Here:

• max_new_tokens=1000: sets the generation limit
• do_sample=False: enables deterministic decoding

Deterministic decoding is useful here because object detection requires stable structured outputs rather than creative variations.

Next, we decode only the generated portion. We then extract the structured JSON response.

The model typically returns a list of detections, so [0] selects the first detection result.

Finally, we return the structured detection dictionary.

def draw_pascal_voc_boxes(i, image, box, label, resize_shape=(1000,1000)):
   dpi = 72
   width, height = image.size
   fig, ax = plt.subplots(1, figsize=[width/dpi, height/dpi], tight_layout={'pad':0})

   ax.imshow(image)

   ymin, xmin, ymax, xmax = box
   re_h, re_w = resize_shape if resize_shape is not None else (height, width)
   xmin = (xmin / re_w) * width
   ymin = (ymin/ re_h) * height
   xmax = (xmax / re_w) * width
   ymax = (ymax/ re_h) * height

   w = xmax - xmin
   h = ymax - ymin

   rect = patches.Rectangle(
       (xmin, ymin),
       w,
       h,
       linewidth=10,
       edgecolor="green",
       facecolor="none"
   )
   ax.add_patch(rect)

   if label is not None:
       ax.text(xmin, ymin-25, label, fontsize=24, bbox=dict(facecolor="yellow", alpha=0.5))

   plt.axis("off")
   plt.savefig(f"boxes_{i}.png")
   plt.close(fig)
   display(fig)

Next, we define the visualization function. This function draws the predicted bounding box on top of the image.

We first create a Matplotlib figure. Then we display the image. Next, we unpack the bounding box coordinates.

Gemma 4 returns normalized coordinates relative to a 1000 × 1000 coordinate grid. We therefore rescale them back to the original image dimensions.

We then compute the bounding box width and height. Next, we create the rectangle overlay. This draws a green bounding box around the detected object. The rectangle is then added onto the image. If a label exists, we also display the object name.

Finally, we save and display the result.

def display_detected_object(image_url, what_object):
   image = load_image(image_url)
   image = resize_to_48_multiple(image)
   detection = detect_object(image_url, what_object)
   box = detection["box_2d"]
   label = detection.get("label", f"{what_object}")
   draw_pascal_voc_boxes("1000", image, box, label)

Next, we define a helper wrapper function. This combines:

• Image loading
• Detection
• Visualization

into one simple pipeline.

We first run object detection. Then extract the returned bounding box and label.

Finally, we visualize the prediction.

display_detected_object("https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/bird.png", "bird")

We can now run the complete pipeline.

Output

At this stage, Gemma 4 identifies the bird in the image, predicts the bounding box coordinates, and overlays the detection visually.

This is the input image:

This is the output image:

This example demonstrates that Gemma 4 is not limited to text generation or captioning. It can also produce structured spatial predictions, enabling workflows such as:

• Object localization
• Visual grounding
• UI element detection
• Document understanding
• Interactive visual reasoning

Image Captioning with Gemma 4 Vision-Language Models

Image captioning is one of the most fundamental multimodal tasks for vision-language models. In this example, we ask Gemma 4 to observe an image and generate a detailed natural language description of the scene.

messages = [
   {
       "role": "user",
       "content": [
           {"type": "image", "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/mosque.jpg"},
           {"type": "text", "text": "Write single detailed caption for this image."},
       ],
   },
]

We first define the multimodal input message. The input contains 2 parts:

• An image
• A text instruction

The image block specifies the image URL. This tells the processor to load the image and prepare it for the vision encoder inside Gemma 4.

Next, we provide the captioning instruction. The wording of the prompt matters here. Since we request a single detailed caption, the model focuses on generating one coherent descriptive sentence instead of multiple fragmented observations.

inputs = processor.apply_chat_template(
   messages,
   tokenize=True,
   return_dict=True,
   return_tensors="pt",
   add_generation_prompt=True,
).to(model.device)

Next, we process the multimodal input.

This step converts the image and text into the structured format expected by Gemma 4.

The processor internally handles:

• Image preprocessing
• Tokenization
• Prompt formatting
• Tensor creation

output = model.generate(**inputs, max_new_tokens=512)

After preprocessing, we generate the caption. Here, max_new_tokens=512 sets the maximum response length. Captioning tasks generally require fewer tokens than HTML generation or reasoning-heavy workflows, but we still allow enough space for detailed descriptions.

input_len = inputs.input_ids.shape[-1]
generated_text_ids = output[0][input_len:]
generated_text = processor.decode(generated_text_ids, skip_special_tokens=True)
result = processor.parse_response(generated_text)

Next, we isolate the generated response. This removes the original prompt tokens and keeps only the newly generated caption. We then decode the generated tokens into readable text. The response is then parsed into a cleaner structure.

print(result["content"])

Finally, we print the generated caption.

Output

At this stage, Gemma 4 analyzes the image and generates a detailed description of the mosque scene.

This is the input image:

This is the generated output:

A bustling outdoor scene features a grand, domed building, likely a historical or religious structure, surrounded by lush greenery, palm trees, and pedestrians walking along a paved walkway. The sky is bright blue with some white clouds, suggesting pleasant weather. In the background, taller minarets are visible, adding to the architectural grandeur of the location. The foreground shows a metal railing and a crowd of people, indicating a popular public space.

Although captioning appears simpler than function calling or video understanding, it is still a strong test of multimodal reasoning. The model must:

• Identify objects
• Understand spatial relationships
• Infer scene context
• Convert visual information into fluent language

This example also highlights the consistency of the Gemma 4 workflow. Whether we perform:

• Screenshot-to-code generation
• Video reasoning
• Function calling
• Captioning

the overall inference pipeline remains nearly identical. Only the input modality and prompt change.

Audio Understanding with Gemma 4

One of the most interesting capabilities of the Gemma 4 E-series models is native audio understanding. Instead of relying on a separate speech recognition model, Gemma 4 can directly process audio inputs and reason about them inside the same multimodal pipeline.

messages = [
   {
       "role": "user",
       "content": [
           {"type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/winning_call.mp3"},
           {"type": "text", "text": "Can you describe this audio in detail?"},
       ],
   },
]

We begin by defining the multimodal input.

The first content block provides the audio input. This tells the processor that the modality is audio rather than image or video.

The second content block contains the text instruction. This prompt asks the model to analyze and explain the audio content in natural language.

inputs = processor.apply_chat_template(
   messages,
   tokenize=True,
   return_dict=True,
   return_tensors="pt",
   add_generation_prompt=True,
).to(model.device)

Next, we process the multimodal input. This step converts the audio and text into tensors that Gemma 4 can process internally.

As discussed in the first part of this series, the E2B and E4B variants include a dedicated audio encoder. The processor handles the conversion of raw audio into the representation expected by this encoder automatically.

output = model.generate(
   **inputs,
   max_new_tokens=1000,
   do_sample=False,
)

After preprocessing, we generate the response.

Here, max_new_tokens=1000 defines the maximum output length for the generated response.

We also use do_sample=False. This disables sampling and makes generation deterministic. Instead of producing varied outputs each time, the model selects the most likely token at every step. For descriptive tasks (e.g., audio understanding), deterministic decoding often produces more stable and reproducible results.

print(processor.decode(output[0], skip_special_tokens=True))

Finally, we decode and print the generated response.

The decode() function converts token IDs back into readable text, while skip_special_tokens=True removes internal formatting tokens.

Output

At this stage, Gemma 4 analyzes the audio clip and generates a detailed natural language description of what it hears.

This is the generated output:

user
Can you describe this audio in detail?
model
Okay, here's a detailed description of the audio you provided:
**Overall Impression:**
The audio is a lively, energetic, and enthusiastic sports commentary, likely from a baseball game. The tone is excited, building anticipation and celebrating a significant moment.
**Specific Details:**
* **Soundscape:** The audio features the sounds of a live sports broadcast. This includes the voice of a commentator, likely with some background noise of a stadium or crowd, though the focus is clearly on the commentary.
* **Commentary Style:** The commentary is fast-paced, dynamic, and uses typical sports jargon and exclamations. The delivery is high-energy and passionate, reflecting the excitement of the game.
* **Key Phrases and Content:**
    * **"And the O1 pitcher on the way to Edgar Martinez swung on the line."**: This sets the scene, indicating a specific play involving a pitcher and a batter.
    * **"Here comes Joey! Here's Ginger in third base!"**: These are calls to action, identifying players and their positions.
    * **"They've got a way to win!"**: This conveys a sense of hope and determination.
    * **"The problem of plate will be late the manner is glaring the fans for the American League championship!"**: This is a more complex sentence, likely referring to a strategic situation and the importance of the championship.
    * **"I don't believe it!"**: An expression of surprise or disbelief, adding to the excitement.
    * **"And just continues my old time!"**: This suggests a continuation of a successful or exciting sequence of events.
* **Emotional Tone:** The tone is overwhelmingly positive, excited, and celebratory. There's a palpable sense of anticipation and triumph.
**In Summary:**
The audio is a segment of a high-energy baseball game broadcast. It captures a moment of intense action, featuring dynamic commentary that builds excitement and highlights the stakes of a championship game. The commentator's enthusiasm is infectious and effectively conveys the thrill of the moment.

This example highlights an important direction in multimodal AI systems. Traditionally, tasks such as the following:

• Speech recognition
• Audio captioning
• Sound event detection
• Spoken question answering

often required separate specialized models.

Gemma 4 instead handles these tasks within a unified multimodal framework, allowing text, images, video, and audio to all flow through the same inference pipeline.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: July 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this lesson, we explored how to build multimodal applications using Gemma 4 and Hugging Face Transformers. Starting from environment setup and model loading, we walked through several real-world multimodal workflows (e.g., including screenshot-to-code generation, video understanding, multimodal function calling, image captioning, audio understanding, and object detection).

One of the biggest takeaways from Gemma 4 is the consistency of its inference pipeline. Whether the input is text, images, video, or audio, the overall workflow remains nearly identical:

• Define the multimodal conversation
• Process the inputs using the processor
• Run generation
• Decode and parse the outputs

This unified design makes multimodal development significantly simpler.

We also saw that Gemma 4 is not limited to conversational AI. It can generate HTML from screenshots, reason over videos, invoke external tools, localize objects with bounding boxes, and understand audio, all inside a single multimodal framework. Together, these capabilities make Gemma 4 a powerful foundation for building next-generation agentic and multimodal AI systems.

Citation Information

Thakur, P. “Building Multimodal AI Applications with Gemma 4 and Transformers,” PyImageSearch, S. Huot, G. Kudriavtsev, and A. Sharma, eds., 2026, https://pyimg.co/09dks

@incollection{Thakur_2026_build-multimodal-ai-apps-w-gemma-4-transformers,
  author = {Piyush Thakur},
  title = {{Building Multimodal AI Applications with Gemma 4 and Transformers}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Georgii Kudriavtsev and Aditya Sharma},
  year = {2026},
  url = {https://pyimg.co/09dks},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Building Multimodal AI Applications with Gemma 4 and Transformers appeared first on PyImageSearch.

Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models

In the previous lesson, we explored the power of Qwen3’s text-only models for building intelligent chatbots capable of deep reasoning. However, the real world is fundamentally multimodal: humans do not just communicate through text; we perceive and interact with the world through vision and other sensory inputs. The ability to understand and reason about visual information alongside text represents a crucial step toward more capable and human-like AI systems. This is where Qwen3-VL (Vision-Language) enters the picture, bringing state-of-the-art multimodal understanding to the open-source community.

Qwen3-VL represents a significant advancement in vision-language models, building upon the robust foundations of the Qwen3 architecture while adding sophisticated visual understanding capabilities. Like its text-only counterpart, Qwen3-VL features both Instruct and Thinking variants, enabling users to choose between fast, direct responses and deep, step-by-step reasoning for complex visual understanding tasks.

The model can process and understand images alongside text queries, making it invaluable for applications (e.g., visual question answering, image captioning, complex scene understanding, and multi-image reasoning). Whether you are analyzing medical images, interpreting charts and diagrams, or building assistive technologies for visually impaired users, Qwen3-VL provides the foundational capabilities needed for production-ready multimodal systems.

In this lesson, we extend our chatbot from Lesson 1 by integrating Qwen3-VL’s vision-language capabilities. We will learn how to handle multimodal inputs where users can upload images alongside their text queries, and the chatbot will respond with contextually appropriate answers that demonstrate understanding of both modalities. Through practical implementation using the Hugging Face Transformers library and Gradio, we will build an interactive web interface that seamlessly processes image-text pairs.

The Thinking mode will reveal the model’s visual reasoning process, showing how it analyzes images step-by-step before arriving at conclusions. By the end of this lesson, you will have a fully functional multimodal chatbot that can engage in sophisticated conversations about visual content, laying the groundwork for even more advanced capabilities we will add in subsequent lessons.

This lesson is the 2nd in a 4-part series on Building with Qwen3 Models:

1. Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models
2. Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models (this tutorial)
3. Lesson 3
4. Lesson 4

To learn about Qwen3 and build intelligent applications from scratch, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Qwen3-VL Vision-Language Model: Architecture, Training, and Capabilities

Qwen3-VL Architecture Overview: SigLIP2 Vision Encoder and Multimodal Transformer Design

Qwen3-VL (Figure 1) builds upon the Qwen3 language model architecture by integrating a powerful vision encoder capable of processing both static images and dynamic video content. The model employs a vision encoder initialized from SigLIP2-SO-400M, containing approximately 543 million parameters specifically dedicated to visual understanding.

This vision encoder is trained on a carefully curated mixture of image and video data, ensuring strong performance across both image understanding tasks and video comprehension scenarios. The integration of vision and language modalities is achieved through an elegant architectural design that preserves the reasoning capabilities of the base language model while adding rich visual understanding.

The Qwen3-VL series offers multiple model sizes to accommodate different computational budgets and application requirements. The lineup includes compact models (e.g., Qwen3-VL-2B) designed for edge deployment and resource-constrained environments, mid-size variants (e.g., Qwen3-VL-4B and Qwen3-VL-8B) that balance performance and efficiency, and larger models up to Qwen3-VL-14B that push the boundaries of multimodal understanding.

Each model variant maintains the same architectural principles but scales the number of layers, attention heads, and hidden dimensions to achieve different capacity levels. All variants support extended context lengths, with smaller models handling 32K tokens and larger models supporting up to 128K tokens, enabling them to process long documents with multiple images or extended video sequences.

A key innovation in Qwen3-VL is the Time-aligned Multimodal Rotary Position Embedding (TM-RoPE), which extends the traditional Multimodal RoPE (M-RoPE) by incorporating explicit temporal information. TM-RoPE factorizes the conventional rotary position embedding into three distinct dimensions: temporal, height, and width. This design enables the model to effectively capture both fine-grained local temporal variations and long-range dependencies across video frames.

For text inputs, the three components share identical position identifiers, making TM-RoPE equivalent to one-dimensional RoPE. For images, a constant temporal ID is assigned to all visual tokens while their spatial positions determine the height and width IDs. For video, each frame receives a monotonically increasing temporal ID adjusted based on actual timestamps, ensuring consistent temporal resolution of 80 milliseconds per ID. This sophisticated positional encoding scheme enables seamless integration of information across diverse modalities.

The vision encoder processes visual inputs by first converting them into a format suitable for the transformer architecture. For images, the encoder divides the input into patches and projects these patches into the model’s hidden dimension. For videos, frames are sampled at a dynamic frame rate to preserve video information while aligning with audio sampling rates in multimodal scenarios. The vision encoder outputs a sequence of visual tokens that are then concatenated with text tokens and processed by the unified transformer backbone. This design allows the model to attend to both visual and textual information jointly, enabling rich cross-modal reasoning. The model can identify objects, understand spatial relationships, read text within images through OCR capabilities, interpret charts and diagrams, and reason about sequences of events in videos.

Both Qwen3-VL-Instruct and Qwen3-VL-Thinking variants are available, mirroring the dual-mode capability of the text-only models. The Instruct variant provides fast, direct responses to visual questions, making it ideal for applications requiring low latency (e.g., real-time image analysis or interactive user interfaces).

The Thinking variant generates explicit reasoning traces that reveal the model’s step-by-step visual analysis process. When presented with a complex image, the Thinking model verbalizes its observations, identifies relevant details, makes connections between visual elements, and systematically works toward a conclusion. This transparency is invaluable for applications requiring explainable AI, educational tools where understanding the reasoning process is important, and debugging or improving system performance.

Training Pipeline: Multimodal Pretraining with Image-Text and Video-Text Data

The training of Qwen3-VL follows a 4-stage approach (Table 1) that builds visual understanding capabilities on top of the strong language foundation established during Qwen3’s pretraining. The training corpus includes diverse sources of image-text pairs and video-text sequences, covering a wide range of domains, styles, and complexity levels.

The data encompasses natural images from everyday scenarios, professional photography, scientific charts and diagrams, screenshots of user interfaces, medical imagery, satellite imagery, artwork and historical photographs, and video sequences capturing dynamic events and actions. This diversity ensures the model develops robust visual understanding that generalizes well to novel scenarios.

The pretraining stage for Qwen3-VL employs a combination of objectives designed to align visual and textual representations. The primary training objective uses a contrastive learning approach similar to CLIP, where the model learns to associate images with their corresponding textual descriptions by maximizing the similarity between matching image-text pairs while minimizing similarity between non-matching pairs.

Additionally, the model is trained on image captioning tasks where it must generate descriptive text given an image, and visual question answering tasks where it must answer questions about image content. These complementary objectives encourage the model to develop both discriminative abilities (understanding what an image depicts) and generative abilities (producing accurate descriptions and responses).

A critical aspect of Qwen3-VL’s training is the integration of visual and language data at an early stage. Rather than training vision and language components separately and then attempting to bridge them, Qwen3-VL employs joint training from the beginning. This approach enables the model to learn shared representations that naturally capture cross-modal relationships.

The training data is carefully balanced to include both unimodal tasks (pure text or pure image understanding) and cross-modal tasks (image-text reasoning, visual question answering). This balance prevents modality-specific performance degradation (a common issue where improving one modality comes at the cost of another) and ensures the model maintains strong capabilities across all input types.

The vision encoder undergoes specialized training to extract meaningful features from images and videos. For static images, the encoder learns to identify salient objects, understand spatial relationships, recognize text through OCR, interpret abstract concepts like emotions or art styles, and extract information from specialized content (e.g., charts, graphs, and diagrams).

For video understanding, the encoder is trained to track objects across frames, understand temporal relationships and event sequences, recognize actions and activities, and maintain consistent object identity throughout clips. This comprehensive training regimen produces visual representations that are rich, detailed, and optimized for downstream reasoning tasks.

Post-training for Qwen3-VL follows a similar philosophy to the text-only models, with instruction tuning and reinforcement learning adapted for multimodal scenarios. The instruction tuning phase uses high-quality human-annotated image-text conversations where the model learns to follow user instructions that reference visual content.

These instructions span diverse task types (e.g., describing images in varying levels of detail, answering specific questions about image content, comparing multiple images, performing visual reasoning and inference, extracting structured information from images, and generating creative content inspired by images). The Thinking variant receives additional training on reasoning traces that explicitly verbalize the visual analysis process, teaching the model to describe what it observes, explain why certain observations are relevant, and show how visual evidence supports its conclusions.

Reinforcement learning fine-tuning (RLHF) further refines the model’s behavior based on human preferences. For multimodal tasks, this involves collecting feedback on the quality, accuracy, and relevance of responses to visual queries.

The model learns to prioritize responses that accurately describe visual content, provide helpful and relevant information, demonstrate appropriate levels of detail based on the query, acknowledge when visual information is ambiguous or unclear, and maintain consistency between visual observations and textual reasoning. For smaller Qwen3-VL models, knowledge distillation from larger variants transfers both visual understanding and reasoning capabilities, enabling compact models to achieve impressive performance despite their reduced parameter count.

Performance Benchmarks: Image, Video, and Multi-Image Reasoning Evaluation

Qwen3-VL demonstrates strong performance across a comprehensive suite of vision-language benchmarks, positioning it as a competitive open-source multimodal model. The model excels in traditional image understanding tasks (e.g., image classification, object detection, and visual question answering), while also showing strong capabilities in more challenging scenarios (e.g., multi-image reasoning, video understanding, and complex visual reasoning).

Across flagship benchmarks for evaluating vision-language models, Qwen3-VL consistently achieves results that rival or surpass much larger proprietary models, demonstrating the effectiveness of its architectural innovations and training methodology.

On general vision-language benchmarks, Qwen3-VL shows strong capabilities across multiple evaluation dimensions. For visual question answering tasks, the model demonstrates strong performance on datasets (e.g., ERQA and RealWorldQA) that require it to answer diverse questions about image content, ranging from simple object recognition to complex reasoning about relationships and attributes. On image captioning benchmarks (e.g., COCO and NoCaps), Qwen3-VL generates accurate, detailed, and fluent descriptions that capture both the main subjects and subtle details in images.

The model’s optical character recognition (OCR) capabilities are particularly strong (Figure 2), enabling it to accurately extract and understand text embedded within images across various fonts, layouts, and contexts.

Multi-image reasoning represents a particularly challenging category where models must integrate information across multiple images to answer questions or perform comparisons. Qwen3-VL excels at these tasks, demonstrating the ability to identify similarities and differences between images, track objects or concepts across a sequence of images, aggregate information from multiple sources, and reason about relationships between distinct visual inputs.

This capability is crucial for applications (e.g., analyzing time-series imagery, comparing before-and-after scenarios, or understanding narratives told through image sequences). The model’s performance on benchmarks specifically designed to test multi-image understanding consistently ranks among the top open-source alternatives.

For video understanding, Qwen3-VL processes sequences of frames to understand temporal dynamics and event structures. The model achieves strong results on video question answering benchmarks where it must comprehend actions, events, and temporal relationships depicted in video clips.

It demonstrates the ability to recognize and track objects throughout videos, understand cause-and-effect relationships between events, identify the sequence and timing of actions, and answer questions requiring temporal reasoning (“What happened before/after X?”). The integration of temporal information through TM-RoPE enables the model to maintain coherent understanding across extended video sequences, a capability that distinguishes it from models that treat videos as disconnected frames.

Specialized benchmarks reveal Qwen3-VL’s capabilities in domain-specific tasks. On mathematical and scientific diagram understanding (e.g., charts, graphs, and geometric figures), the model demonstrates the ability to extract numerical information from visualizations, understand mathematical notation and symbols, reason about spatial relationships in geometry problems, and interpret scientific diagrams and schematics.

For document understanding tasks (Figure 3), including complex layouts with mixed text and images, Qwen3-VL shows strong performance in extracting information from receipts, invoices, and forms, understanding document structure and hierarchy, reading tables and structured data, and processing multi-page documents with consistent context.

When comparing Instruct and Thinking variants, the Thinking model demonstrates particular advantages on tasks requiring complex reasoning or multi-step analysis. While the Instruct variant provides faster responses suitable for straightforward queries, the Thinking variant’s explicit reasoning traces lead to improved accuracy on challenging problems.

Users can observe the model systematically analyzing visual information, identifying relevant details, making intermediate inferences, and arriving at well-justified conclusions. This transparency not only improves performance on difficult tasks but also enables debugging, validation, and trust-building in deployment scenarios.

The Thinking variant shows the largest performance improvements over Instruct on tasks involving visual reasoning chains, ambiguous or complex images requiring careful analysis, questions that require synthesizing information from multiple parts of an image, and scenarios where explainability is valued alongside accuracy (Figure 4).

Implementation: Building a Multimodal Chatbot with Qwen3-VL and Gradio

Now, let us build on our foundation from Lesson 1 to create a multimodal chatbot that can understand and reason about images alongside text. We will integrate Qwen3-VL models and extend the Gradio interface to support image uploads.

Step 1: Installing Dependencies for Qwen3-VL and Multimodal Processing (PyTorch, Transformers, Gradio)

In addition to the packages from Lesson 1, we need a few more libraries for handling images and multimodal processing.

!pip install torch transformers accelerate gradio
!pip install flash-attn --no-build-isolation
!pip install qwen-vl-utils pillow
print("\n✅ Dependencies installed successfully!")

On Line 1, we install the same core dependencies as Lesson 1: PyTorch, Transformers, Accelerate, and Gradio. On Line 2, we include Flash Attention for optimized inference. On Line 3, we add 2 new packages: qwen-vl-utils provides utilities for processing vision inputs specific to Qwen3-VL models, including helper functions for formatting multimodal conversations, and pillow is Python’s standard image-processing library for loading, manipulating, and displaying images. These additional dependencies enable our chatbot to seamlessly handle visual inputs alongside text. On Line 4, we print a confirmation message.

Step 2: Importing Required Libraries

We extend our imports from Lesson 1 to include image processing capabilities.

import torch
from transformers import pipeline
import gradio as gr
from typing import List, Tuple, Optional
from PIL import Image
import requests
from io import BytesIO
from qwen_vl_utils import process_vision_info
from IPython.display import display, Markdown

print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"CUDA device: {torch.cuda.get_device_name(0)}")

On Lines 1-9, we import all necessary modules for our multimodal application. Beyond the basic imports from Lesson 1, we add Image from PIL for image manipulation, requests and BytesIO for downloading and handling image data from URLs, and process_vision_info from qwen_vl_utils for preprocessing vision inputs in the format expected by Qwen3-VL models. The IPython display imports help us visualize images and formatted text in Jupyter notebooks. On Lines 11-14, we perform the same environment checks as before to verify PyTorch installation and GPU availability.

Step 3: Initializing Qwen3-VL Instruct and Thinking Image-Text-to-Text Pipelines

Now we initialize the Qwen3-VL pipelines that can process both images and text.

print("=" * 60)
print("Initializing Qwen3-VL-2B-Instruct and Qwen3-VL-2B-Thinking Text Generation Pipelines")
print("=" * 60)

qwen3vl_instruct = pipeline("image-text-to-text", model="Qwen/Qwen3-VL-2B-Instruct")
qwen3vl_thinking = pipeline("image-text-to-text", model="Qwen/Qwen3-VL-2B-Thinking")

print("\n✅ Pipelines initialized successfully!")

On Lines 1-3, we print informative messages to track initialization progress. On Lines 5 and 6, we initialize 2 pipelines using the "image-text-to-text" task type instead of "text-generation" from Lesson 1. This task type indicates that the pipeline accepts both images and text as input and generates text as output.

We use the Qwen3-VL-2B variants, which offer excellent performance while remaining lightweight enough for efficient inference. The models automatically download from Hugging Face if not cached locally (approximately 4-5 GB each for the 2B parameter versions). These pipelines will handle all the complexity of encoding images, combining them with text, and generating appropriate responses.

Step 4: Testing Multimodal Conversations with Image Uploads and Text Prompts

Let us test the models with a conversation that includes images to understand their multimodal capabilities.

# Initialize conversation history
conversation = []

# First message with image
img1 = Image.open(BytesIO(requests.get("https://media.istockphoto.com/photos/dogs-playing-picture-id481634156", timeout=10).content))
user_msg_1 = "Describe the image briefly."
conversation.append({"role": "user", "content": [{"type": "text", "text": user_msg_1}, {"type": "image", "image": img1}]})
display(Markdown(f"**User**: {user_msg_1}"))
display(img1)

response_1 = qwen3vl_instruct(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
response_1 = response_1[0]['generated_text']
display(Markdown(f"\n**Assistant**: {response_1}\n"))

# Update history
conversation.append({"role": "assistant", "content": [{"type": "text", "text": response_1}]})

# Second message with different image (continuation)
img2 = Image.open(BytesIO(requests.get("https://th.bing.com/th/id/OIP.i7RVdGQWnsiTssgbUeCmsAHaLG", timeout=10).content))
user_msg_2 = "Differentiate between first and second image."
conversation.append({"role": "user", "content": [{"type": "text", "text": user_msg_2}, {"type": "image", "image": img2}]})
display(Markdown(f"**User**: {user_msg_2}"))
display(img2)

response_2 = qwen3vl_thinking(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
trace = response_2[0]['generated_text'].split('</think>')[0]
response_2 = response_2[0]['generated_text'].split('</think>')[1]
response_2 = (
    "<details><summary>Show trace</summary>\n\n"
    "```text\n" + (trace or "").strip() + "\n```\n"
    "</details>\n\n"
    + (response_2 or "")
)
display(Markdown(f"\nAssistant: {response_2}"))

On Line 2, we initialize an empty conversation list as before. On Lines 5-7, we download an image from a URL using the requests library, load it into a PIL Image object, and create the first user message. Note the new format on Line 7: instead of a simple string, the content is now a list containing both text and image dictionaries. This structure allows us to include multiple modalities in a single message. On Lines 8 and 9, we display both the text query and the image for visibility.

On Lines 11-13, we generate a response using the Instruct model with the same parameters as Lesson 1. The model processes both the text query and the image, generating a description that demonstrates visual understanding. On Line 16, we update the conversation history with the assistant’s response, formatting it as a text-type content entry.

On Lines 19-23, we introduce a second image and ask a comparative question that requires the model to recall information from the first image while analyzing the second. This tests the model’s ability to maintain visual context across conversation turns. On Lines 25-33, we switch to the Thinking model for the response. As in Lesson 1, we parse the thinking trace from the final answer and format it as collapsible HTML. The thinking trace now reveals not just textual reasoning but also how the model analyzes visual information: describing what it observes in each image, identifying key differences, and building toward a coherent comparative answer.

Figure 5 shows the output of the code snippet above, demonstrating the reasoning capabilities of both models.

Step 5: Building a Gradio Multimodal Chat Interface with Image Input Support

Now we will build a comprehensive Gradio interface that supports image uploads alongside text input.

import base64

def create_chatbot_interface(chatbot, thinking_chatbot):
    """
    Gradio interface for multimodal chatbot interaction
    """

    def pil_to_base64(img):
        buf = BytesIO()
        img.save(buf, format="PNG")
        return "data:image/png;base64," + base64.b64encode(buf.getvalue()).decode()

    def add_user_message(message: str, image, conversation, history):
        if not message.strip():
            return history, ""
        if history is None:
            history = []
        conversation.append({"role": "user", "content": [{"type": "text", "text": message}]})
        if image is not None:
            conversation.append({"role": "user", "content": [{"type": "image", "image": image}]})

        display_text = message or ""
        if image is not None:
            img_str = pil_to_base64(image)
            display_text += f"\n\n![uploaded image]({img_str})"
        history.append([display_text, None])

        return history, "", None   # show user message immediately

On Line 1, we import the base64 module for encoding images as strings. On Lines 3-6, we define our main interface function with a docstring explaining its purpose. On Lines 8-11, we define a helper function pil_to_base64 that converts PIL images to base64-encoded strings. This is necessary because we want to display uploaded images directly in the chat history. The function resizes images to 256×256 pixels for efficient display and encodes them as PNG data URLs.

On Lines 13-28, we define the add_user_message function, which now handles both text and optional image inputs. On Lines 14-17, we validate that the message is not empty and initialize history if needed. On Lines 18-20, we append the text content to the conversation and, if an image is provided, append it as a separate content entry with type "image".

This format matches what Qwen3-VL expects. On Lines 22-26, we create a display string for the chat history. If an image was uploaded, we convert it to a base64 data URL and embed it using markdown image syntax, allowing Gradio to render it directly in the chatbot UI. On Line 28, we return the updated history, clear the text input, and clear the image input, providing immediate visual feedback.

def chat_function(message: str, conversation, history, model_choice: str):
        try:
            if model_choice == "Thinking":
                response = thinking_chatbot(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
                trace = response[0]['generated_text'].split('</think>')[0]
                response = response[0]['generated_text'].split('</think>')[1]
                assistant_text = (
                    "<details><summary>Show thinking trace</summary>\n\n"
                    "```text\n" + (trace or "").strip() + "\n```\n"
                    "</details>\n\n"
                    + (response or "")
                )
            else:
                response = chatbot(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
                response = response[0]['generated_text']
                assistant_text = response or ""

            conversation.append({"role": "assistant", "content": [{"type": "text", "text": assistant_text}]})
            history[-1][1] = assistant_text
            return history, "", None
        except Exception as e:
            history[-1][1] = f"Exception: {e}"
            return history, "", None

On Lines 1-23, we define the core chat function that generates responses. The logic is nearly identical to Lesson 1, but now it handles multimodal conversation contexts. On Lines 3-12, if Thinking Mode is selected, we generate a response with reasoning traces and format them as collapsible details. On Lines 13-16, if Instruct Mode is selected, we generate a direct response. The models automatically process any images in the conversation context without us needing to explicitly pass them again, since the conversation list maintains the complete multimodal history. On Lines 18-23, we update the conversation with the assistant’s response and handle any exceptions gracefully.

    with gr.Blocks() as interface:
        gr.Markdown("## 🤖 Qwen3-VL-2B Multimodal Chatbot with Deep Thinking")

        with gr.Row():
            chatbot_ui = gr.Chatbot(scale=4, height=1000)

        with gr.Row():
            msg = gr.Textbox(placeholder="Type your message here...", scale=8)
            image_input = gr.Image(
                label="Upload Image (optional)",
                type="pil",
                scale=1
            )
            send_btn = gr.Button("📤 Send", scale=1)
            stop_btn = gr.Button("⏹️ Stop", scale=1)
            model_selector = gr.Dropdown(
                choices=["Instruct", "Thinking"],
                value="Instruct",
                label="Model selection",
                scale=2
            )

        history_state = gr.State([])
        conversation_state = gr.State([])

        # Step 1: Show user message immediately
        send_btn.click(
            add_user_message,
            inputs=[msg, image_input, conversation_state, history_state],
            outputs=[chatbot_ui, msg, image_input],
        )

        # Step 2: Then process with chatbot
        send_event = send_btn.click(
            chat_function,
            inputs=[msg, conversation_state, history_state, model_selector],
            outputs=[chatbot_ui, msg, image_input],
        )

        stop_btn.click(None, None, None, cancels=[send_event])

    return interface

On Lines 1 and 2, we create the Gradio interface with an updated title reflecting multimodal capabilities. On Lines 4 and 5, we create the chatbot display area. On Lines 7-21, we build the input row with several components: a text box for messages (Line 8), an image upload widget that accepts PIL images (Lines 9-13), send and stop buttons (Lines 14 and 15), and a model selector dropdown (Lines 16-21). The type="pil" parameter on the image input ensures uploaded images are automatically converted to PIL Image objects.

On Lines 23 and 24, we initialize state variables for tracking conversation history and display state. On Lines 27-31, we configure the send button’s first action to immediately display the user’s message and uploaded image. Note that image_input is now included in both inputs and outputs, allowing us to clear it after sending. On Lines 34-38, we configure the send button’s second action to process the message with the selected model. On Line 40, we wire up the stop button to cancel ongoing generation.

Step 6: Launching the Qwen3-VL Multimodal Chatbot Application

Finally, we create and launch our multimodal chatbot interface.

# Create the interface
interface = create_chatbot_interface(qwen3vl_instruct, qwen3vl_thinking)
print("✅ Gradio interface created!")

print("Launching Gradio interface...")
print("Access the chatbot at: http://localhost:7860")
print("Press the 'Stop' button in the output to stop the server\n")

interface.launch()

On Line 2, we instantiate the interface by passing both VL model pipelines to our creation function. On Lines 5-7, we print informative messages about accessing the application. On Line 9, we launch the Gradio interface on the default port. Users can now interact with a fully multimodal chatbot that understands and reasons about both text and images.

They can upload images individually or alongside text queries, ask questions about uploaded images, request comparisons between multiple images across the conversation, and switch between Instruct and Thinking Modes to see how the model analyzes visual information. The interface provides a seamless, intuitive way to explore Qwen3-VL’s powerful multimodal capabilities.

Multimodal Chatbot Results: Instruct Mode vs Thinking Mode Analysis

To illustrate how our chatbot adapts between Instruct Mode and Thinking Mode, we tested it on 2 different multimodal tasks. Below, we start with the concise instructive response, then show how the same system expands into deeper reasoning when Thinking Mode is enabled.

In Figure 6, we tried the chatbot in Instruct Mode, where it delivered a clear and balanced summary: sales across America, EMEA (Europe, the Middle East, and Africa), and Asia increased steadily over 4 years, with the strongest growth in Year 4. It also provided exact figures per region and year, but kept the explanation concise enough to avoid overwhelming detail.

This demonstrates how Instruct Mode excels at giving direct, useful insights from visual inputs, ideal for users who want quick answers.

In Figure 7, we uploaded a cartoon-style puzzle showing 2 forest scenes and asked the chatbot to “Spot the difference between both the images. Answer briefly.”

In Thinking Mode, the chatbot does not just state the differences; it walks through the reasoning: identifying the animals in each scene, comparing them, and concluding that the bottom image contains a zebra and an orange bird instead of the pink flamingo in the top image.

This shows how Thinking Mode provides step-by-step transparency, making the reasoning process visible and trustworthy.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: July 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

Throughout this lesson, we have explored Qwen3-VL and its remarkable multimodal capabilities that enable AI systems to understand and reason about visual information alongside text. We began by understanding the architectural innovations that make Qwen3-VL possible, including the SigLIP2-based vision encoder with 543 million parameters, the sophisticated TM-RoPE positional encoding that seamlessly integrates temporal, height, and width dimensions, and the joint training approach that prevents modality-specific performance degradation.

We learned how the model processes images by dividing them into patches, videos by sampling frames at dynamic rates, and maintains consistent understanding across extended sequences through absolute temporal encoding. The availability of both Instruct and Thinking variants provides flexibility between fast responses and transparent, step-by-step visual reasoning.

We then translated this theoretical knowledge into a practical implementation by extending our chatbot from Lesson 1 with multimodal capabilities. Through hands-on coding, we learned how to install and import additional dependencies for image processing, initialize vision-language pipelines using the “image-text-to-text” task type, structure multimodal conversations with content lists containing both text and image entries, and create a Gradio interface that supports image uploads alongside text input.

The implementation demonstrated key patterns for multimodal development: embedding images as base64 strings for display, maintaining visual context across conversation turns, parsing and displaying thinking traces for visual reasoning, and handling the complexity of mixed-modality inputs with clean, modular code. We saw firsthand how the Thinking model verbalizes its visual analysis, describing observations, identifying relevant details, and building toward well-reasoned conclusions.

This lesson represents a crucial step forward in our journey to build a comprehensive AI assistant. We have progressed from text-only interaction to sophisticated multimodal understanding, enabling applications such as visual question answering, image description and captioning, multi-image comparison and analysis, and video understanding through temporal reasoning.

In the next lesson (Lesson 3) of our series, we will add Qwen3-Guard to introduce safety moderation that protects users from harmful content in both text and image inputs. Following that (Lesson 4), we will integrate Qwen3-Embedding to enable retrieval-augmented generation, allowing our chatbot to search through knowledge bases and ground its responses in retrieved information. By combining these components (language understanding, visual reasoning, safety checks, and knowledge retrieval), we are building toward a production-ready multimodal AI assistant that demonstrates the full potential of the Qwen3 ecosystem.

Citation Information

Mangla, P. “Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/jf6oe

@incollection{Mangla_2026_building-multimodal-chatbot-qwen3-vl-instruct-thinking-models,
  author = {Puneet Mangla},
  title = {{Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/jf6oe},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models appeared first on PyImageSearch.

Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models

The landscape of artificial intelligence has witnessed remarkable progress in recent years, with large language models (LLMs) pushing the boundaries of what machines can understand and generate. Among the latest breakthroughs in this field is Qwen3, a comprehensive series of open-weight large language models that achieves state-of-the-art performance across diverse tasks and domains. Qwen3 represents a significant leap forward from its predecessor, Qwen2.5, introducing revolutionary features that fundamentally change how we interact with AI systems.

What sets Qwen3 apart is its unique dual-mode architecture that seamlessly integrates both thinking and non-thinking modes within a single model. This innovation eliminates the need to switch between different models for tasks requiring different levels of reasoning, such as moving from Qwen2.5 to QwQ for complex problem-solving.

The thinking mode enables deep, step-by-step reasoning through extended chain-of-thought (CoT) processes, making it ideal for challenging mathematical problems, complex coding tasks, and multi-step reasoning scenarios. Meanwhile, the non-thinking mode provides fast, direct responses for simpler queries where extensive deliberation is not necessary. This flexibility, combined with thinking budgets that allow fine-grained control over computational resources, makes Qwen3 a versatile tool for real-world applications.

In this lesson, we embark on an exciting journey to build an intelligent chatbot that leverages both Qwen3-Instruct and Qwen3-Thinking models. Through hands-on implementation, we will explore how these models can be integrated into a practical application using Gradio, a powerful framework for creating interactive web interfaces.

By the end of this tutorial, you will understand not only the theoretical underpinnings of Qwen3’s architecture and training methodology but also how to deploy these models in production-ready applications. This lesson serves as the foundation for our multi-part series on building a comprehensive, safe, and multimodal AI assistant using the complete Qwen3 model ecosystem.

This lesson is the 1st in the 4-part series on Building with Qwen3 Models:

1. Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models (this tutorial)
2. Lesson 2
3. Lesson 3
4. Lesson 4

To learn about Qwen3 and build intelligent applications from scratch, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Qwen3 Model Details

Qwen3 Model Architecture: Dense Models, MoE Variants, and Dual-Mode Reasoning

The Qwen3 series represents a comprehensive family of language models, offering both dense and Mixture-of-Experts (MoE) architectures to accommodate diverse computational requirements and use cases. The lineup includes 6 dense models ranging from the compact Qwen3-0.6B to the substantial Qwen3-32B, alongside two powerful MoE variants: Qwen3-30B-A3B and the flagship Qwen3-235B-A22B. The flagship model, with its 235 billion total parameters and 22 billion activated parameters per token, demonstrates the perfect balance between high performance and efficient inference through its sparse activation pattern.

The architecture of Qwen3 builds upon proven foundations while introducing critical enhancements for stability and performance. The dense models share core architectural elements with Qwen2.5, including Grouped Query Attention (GQA) for efficient attention computation, SwiGLU activation functions for better gradient flow, Rotary Positional Embeddings (RoPE) for superior position encoding, and RMSNorm with pre-normalization for training stability. However, Qwen3 introduces 2 significant improvements: the removal of QKV-bias that was present in Qwen2, and the introduction of QK-Norm to the attention mechanism, which ensures stable training especially for larger models.

The model variants span a wide range of sizes to meet different deployment scenarios (Table 1):

For the MoE models, Qwen3 adopts fine-grained expert segmentation with 128 total experts and 8 activated experts per token. Unlike Qwen2.5-MoE, the new design excludes shared experts and instead employs a global-batch load-balancing loss to encourage expert specialization (Table 2).

This architectural innovation has yielded substantial improvements in model performance across downstream tasks. The MoE models achieve performance comparable to much larger dense models while using only a fraction of the activated parameters, making them highly cost-effective for training and inference.

All Qwen3 models utilize Qwen’s tokenizer, which implements byte-level byte-pair encoding (BBPE) with a vocabulary size of 151,669 tokens. This comprehensive vocabulary enables efficient encoding across the 119 languages and dialects supported by Qwen3, from high-resource languages like English and Chinese to low-resource languages and specialized domains.

The combination of architectural innovations, diverse model sizes, and extensive multilingual support makes Qwen3 one of the most versatile and powerful open-source language model families available today.

Qwen3 Training Data and Post-Training Pipeline for Reasoning and Instruction Following

The training of Qwen3 represents a massive undertaking in data curation, processing, and model optimization. The pre-training dataset consists of approximately 36 trillion tokens (double the scale of Qwen2.5) and covers an unprecedented 119 languages and dialects compared to just 29 in the previous version. This expansion includes high-quality content spanning coding, STEM (Science, Technology, Engineering, and Mathematics), reasoning tasks, books, multilingual texts, and synthetic data, ensuring comprehensive knowledge coverage across domains.

To expand the pre-training corpus, the Qwen team employed innovative data augmentation strategies. First, they leveraged the Qwen2.5-VL model to perform text recognition on large volumes of PDF-like documents, extracting trillions of tokens from previously inaccessible sources. The recognized text was then refined using the Qwen2.5 model to improve quality.

Additionally, they employed specialized models (Qwen2.5-Math for mathematical content and Qwen2.5-Coder for code generation) to synthesize trillions of text tokens in various formats, including textbooks, question-answering pairs, instructions, and code snippets across dozens of domains. This synthetic data generation approach allowed for precise control over data quality and diversity.

A key innovation in Qwen3’s training is the development of a multilingual data annotation system that enhanced both quality and diversity. This system annotated over 30 trillion tokens across multiple dimensions such as educational value, fields, domains, and safety.

Unlike previous studies that optimized data mixture at the source or domain level, Qwen3’s approach optimized the data mixture at the instance level through extensive ablation experiments on small proxy models with fine-grained data labels. This granular approach to data curation significantly improved the effectiveness of the training process.

The pre-training process follows a carefully designed 3-stage strategy:

Stage 1. General Stage (S1): All Qwen3 models are trained on over 30 trillion tokens using a sequence length of 4,096 tokens. This stage establishes a strong foundation in language proficiency and general world knowledge across all 119 supported languages and dialects. The models learn fundamental patterns, linguistic structures, and broad knowledge that serves as the basis for more specialized capabilities.

Stage 2. Reasoning Stage (S2): To enhance reasoning abilities, the training corpus is optimized by increasing the proportion of STEM, coding, reasoning, and synthetic data. Models are further trained on approximately 5 trillion high-quality tokens at a sequence length of 4,096 tokens. The learning rate decay is accelerated during this stage to fine-tune the models’ reasoning capabilities, particularly for complex problem-solving tasks in mathematics, science, and programming.

Stage 3. Long Context Stage: In the final pre-training stage, high-quality long context corpora are used to extend the context length from 4,096 to 32,768 tokens. The long context corpus includes 75% of text between 16,384 and 32,768 tokens in length, and 25% of text between 4,096 and 16,384 tokens. The base frequency of RoPE is increased from 10,000 to 1,000,000 using the ABF technique, and YARN and Dual Chunk Attention (DCA) techniques are introduced to achieve a 4-fold increase in sequence length capacity during inference, enabling the models to handle documents up to 128K tokens.

Post-training (Figure 1) employs a sophisticated multi-stage approach to align the models with human preferences and downstream applications. The first 2 stages focus on developing strong reasoning abilities through long chain-of-thought (CoT) cold-start fine-tuning and reinforcement learning, primarily targeting mathematics and coding tasks.

In the final 2 stages, data with and without reasoning paths are combined into a unified dataset for further fine-tuning, enabling the model to handle both thinking and non-thinking modes effectively. General-domain reinforcement learning is then applied to improve performance across a wide range of downstream tasks.

For smaller models, strong-to-weak distillation leverages both off-policy and on-policy knowledge transfer from larger models, significantly outperforming direct reinforcement learning in both performance and training efficiency.

Qwen3 Benchmark Performance: Thinking Budgets, Reasoning, Coding, and STEM Tasks

Qwen3 demonstrates exceptional performance across a comprehensive suite of benchmarks, establishing new standards for open-source language models. The flagship model (Qwen3-235B-A22B) consistently outperforms alternatives across most benchmarks (Table 3).

When evaluated against DeepSeek-V3-Base, which has approximately three times more parameters (671B vs 235B) and nearly double the activated parameters (37B vs 22B), Qwen3-235B-A22B achieves superior results on 14 out of 15 evaluation benchmarks. This includes substantial advantages on general tasks like MMLU (87.81 vs 87.19), MMLU-Redux (87.40 vs 86.14), and BBH (88.87 vs 86.22), as well as specialized domains such as MATH (71.84 vs 62.62) and EvalPlus coding benchmarks (77.60 vs 63.75).

The performance advantages extend across the entire Qwen3 family. The MoE models demonstrate exceptional efficiency, achieving similar performance to dense models with only 1/5 of the activated parameters. For instance, experimental results show that Qwen3-30B-A3B (with only 3B activated parameters) performs comparably to much larger dense models, while Qwen3-235B-A22B outperforms the Qwen2.5 MoE base models with less than 1/2 of their activated parameters and fewer total parameters.

The dense Qwen3 models also show impressive scaling efficiency. Qwen3-1.7B/4B/8B/14B/32B-Base achieve performance comparable to Qwen2.5-3B/7B/14B/32B/72B-Base, respectively, effectively matching models that are 1.5-2x larger in parameter count (Table 4).

On STEM, coding, and reasoning benchmarks, Qwen3 dense base models frequently surpass their Qwen2.5 counterparts at higher parameter scales. For example, Qwen3-32B-Base scores 65.54 on MMLU-Pro compared to Qwen2.5-72B-Base’s 58.07, demonstrating substantial improvements in reasoning capabilities despite having less than half the parameters.

A particularly noteworthy finding is that increasing the thinking budget for thinking tokens leads to consistent improvements in model performance across various tasks (Figure 2). This demonstrates the effectiveness of the dual-mode architecture, where users can dynamically control the trade-off between response speed and reasoning depth based on their specific needs.

The combination of architectural innovations, extensive training data, and sophisticated optimization techniques positions Qwen3 as one of the most capable and efficient open-source language model families available, offering performance competitive with proprietary alternatives while maintaining the benefits of transparency and community-driven development.

Implementation: Building a Deep Thinking Chatbot with Qwen3 and Gradio

Now that we understand the theoretical foundations of Qwen3, let us dive into building a practical chatbot application that leverages both the Instruct and Thinking modes. We will create an interactive web interface using Gradio that allows users to seamlessly switch between fast responses and deep reasoning.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Step 1: Install Qwen3 Chatbot Dependencies with PyTorch, Transformers, and Gradio

First, we need to install the required Python packages for our chatbot. This includes PyTorch for model inference, Transformers for loading the Qwen3 models, and Gradio for building the web interface.

!pip install torch transformers accelerate gradio
!pip install flash-attn --no-build-isolation
print("\n✅ Dependencies installed successfully!")

On Lines 1-3, we install the core dependencies for our chatbot application. The torch package provides the deep learning framework for running model inference, while transformers from Hugging Face gives us easy access to pre-trained Qwen3 models.

The accelerate library optimizes model loading and inference across different hardware configurations, and gradio enables us to build an interactive web interface with minimal code. We also install flash-attn for optimized attention mechanisms that significantly speed up inference, particularly for long sequences. The --no-build-isolation flag ensures compatibility with the existing environment.

Step 2: Importing Required Libraries

Next, we import the necessary libraries and verify our environment setup.

import torch
from transformers import pipeline
import gradio as gr
from typing import List, Tuple, Optional
from IPython.display import display, HTML, Markdown

print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"CUDA device: {torch.cuda.get_device_name(0)}")

On Lines 1-5, we import the essential modules for our application. The torch import provides access to PyTorch functionality, while pipeline from transformers simplifies model loading and inference. Gradio (gr) will power our web interface, and the typing hints ensure code quality. The IPython display utilities help us render formatted output in Jupyter notebooks.

On Lines 7-10, we perform environment checks to verify that PyTorch is properly installed and to confirm whether GPU acceleration is available. If CUDA is available, we also print the GPU device name to ensure we are utilizing hardware acceleration for faster inference.

Step 3: Initialize Qwen3 Instruct and Qwen3 Thinking Text-Generation Pipelines

Now we create instances of both the Instruct and Thinking model pipelines. The pipeline abstraction from Hugging Face makes it incredibly easy to load and use these models.

print("=" * 60)
print("Initializing Qwen3-4B-Instruct and Qwen3-4B-Thinking Text Generation Pipelines")
print("=" * 60)

qwen3_instruct = pipeline("text-generation", model="Qwen/Qwen3-4B-Instruct-2507")
qwen3_thinking = pipeline("text-generation", model="Qwen/Qwen3-4B-Thinking-2507")

print("\n✅ Pipelines initialized successfully!")

On Lines 1-3, we print informative messages to track the initialization process. On Lines 5 and 6, we initialize 2 separate pipelines: one for the Instruct model and one for the Thinking model.

The pipeline function automatically handles model downloading (approximately 8GB for the 4B parameter model), tokenization, and inference setup. We use the Qwen3-4B variants, which offer an excellent balance between performance and resource requirements, making them suitable for a wide range of hardware configurations. The models are automatically downloaded from the Hugging Face model hub if they are not already cached locally.

Step 4: Test Qwen3 Instruct vs. Thinking Models in a Multi-Turn Conversation

Before building the full interface, let us test both models with a simple conversation to understand their behavior.

# Initialize conversation history
conversation = []

# First message
user_msg_1 = "What is machine learning? Explain shortly."
conversation.append({"role": "user", "content": user_msg_1})
display(Markdown(f"**User**: {user_msg_1}"))

response_1 = qwen3_instruct(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
response_1 = response_1[0]['generated_text']
display(Markdown(f"\n**Assistant**: {response_1}\n"))

# Update history
conversation.append({"role": "assistant", "content": response_1})

# Second message (continuation)
user_msg_2 = "Can you give me a simple short example?"
conversation.append({"role": "user", "content": user_msg_2})
display(Markdown(f"**User**: {user_msg_2}"))

response_2 = qwen3_thinking(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
trace = response_2[0]['generated_text'].split('</think>')[0]
response_2 = response_2[0]['generated_text'].split('</think>')[1]
response_2 = (
    "<details><summary>Show trace</summary>\n\n"
    "```text\n" + (trace or "").strip() + "\n```\n"
    "</details>\n\n"
    + (response_2 or "")
)
display(Markdown(f"\nAssistant: {response_2}"))

On Lines 1 and 2, we initialize an empty conversation list that will store the dialogue history in the format expected by the models. On Lines 4-7, we create the first user message asking about machine learning and append it to the conversation with the “user” role.

On Lines 9 and 10, we use the Instruct model to generate a response with do_sample=False for deterministic output and max_new_tokens=1024 to limit response length. The return_full_text=False parameter ensures we only get the newly generated text, not the entire prompt.

On Lines 13-18, we update the conversation history with the assistant’s response and then add a follow-up question. This demonstrates how the models maintain context across multiple turns. On Line 21, we switch to the Thinking model for the second response, which generates output with explicit reasoning traces.

On Lines 22-29, we parse the model’s output to separate the thinking process (enclosed in <think> tags) from the final answer. We format this as collapsible HTML details, allowing users to optionally view the model’s step-by-step reasoning process. This separation showcases the key difference between Instruct and Thinking modes: while Instruct provides direct answers, Thinking reveals its internal reasoning process.

Figure 3 shows the output of the above code snippet, demonstrating the reasoning capabilities of both models.

Step 5: Creating the Interactive Gradio Interface

Now we will build a comprehensive web interface that allows users to interact with both models through a clean, intuitive chat interface.

def create_chatbot_interface(chatbot, thinking_chatbot):
    """
    Gradio interface for chatbot interaction
    """

    def add_user_message(message: str, conversation, history):
        if not message.strip():
            return history, ""
        if history is None:
            history = []
        conversation.append({"role": "user", "content": message})
        history.append([message, None])
        return history, ""   # show user message immediately

    def chat_function(message: str, conversation, history, model_choice: str):
        try:
            if model_choice == "Thinking":
                response = thinking_chatbot(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
                trace = response[0]['generated_text'].split('</think>')[0]
                response = response[0]['generated_text'].split('</think>')[1]
                assistant_text = (
                    "<details><summary>Show thinking trace</summary>\n\n"
                    "```text\n" + (trace or "").strip() + "\n```\n"
                    "</details>\n\n"
                    + (response or "")
                )
            else:
                response = chatbot(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
                response = response[0]['generated_text']
                assistant_text = response or ""

            conversation.append({"role": "assistant", "content": assistant_text})
            history[-1][1] = assistant_text
            return history, ""
        except Exception as e:
            history[-1][1] = f"Exception: {e}"
            return history, ""

On Lines 1-4, we define the main function that creates our chatbot interface, taking both the Instruct and Thinking model pipelines as parameters. On Lines 6-13, we define add_user_message, a helper function that handles user input. It validates that the message is not empty, initializes the history if needed, and appends the user’s message to both the conversation state and the display history. The function returns the updated history with the user’s message immediately visible, providing responsive feedback.

On Lines 15-37, we define the chat_function that processes messages and generates responses. The function first checks the model_choice parameter to determine which model to use. On Lines 17-26, if “Thinking” mode is selected, we generate a response using the thinking model and parse out the reasoning trace, formatting it as collapsible HTML for optional viewing.

On Lines 27-30, if “Instruct” mode is selected, we generate a straightforward response without thinking traces. On Lines 32-37, we update the conversation history with the assistant’s response and handle any exceptions gracefully by displaying error messages.

 with gr.Blocks() as interface:
        gr.Markdown("## 🤖 Qwen3-4B Chatbot with Deep Thinking")

        with gr.Row():
            chatbot_ui = gr.Chatbot(scale=4)

        with gr.Row():
            msg = gr.Textbox(placeholder="Type your message here...", scale=8)
            send_btn = gr.Button("📤 Send", scale=1)
            stop_btn = gr.Button("⏹️ Stop", scale=1)
            model_selector = gr.Dropdown(
                choices=["Instruct", "Thinking"],
                value="Instruct",
                label="Model selection",
                scale=2
            )

        history_state = gr.State([])
        conversation_state = gr.State([])

        # Step 1: Show user message immediately
        send_btn.click(
            add_user_message,
            inputs=[msg, conversation_state, history_state],
            outputs=[chatbot_ui, msg],
        )

        # Step 2: Then process with chatbot
        send_event = send_btn.click(
            chat_function,
            inputs=[msg, conversation_state, history_state, model_selector],
            outputs=[chatbot_ui, msg],
        )

        stop_btn.click(None, None, None, cancels=[send_event])

    return interface

On Lines 1 and 2, we use Gradio’s Blocks API to create a custom layout, starting with a markdown header. On Lines 4 and 5, we create the main chatbot display area with scale=4 to give it more screen space. On Lines 7-16, we build the input row containing a text box for user messages, send and stop buttons, and a dropdown menu for model selection. The scaling parameters control the relative widths of these elements for an optimal layout.

On Lines 18 and 19, we initialize 2 state variables: history_state for the chat display and conversation_state for the full conversation context needed by the models. On Lines 21-26, we configure the send button’s first action to immediately display the user’s message, providing instant feedback.

On Lines 28-33, we configure the send button’s second action to process the message with the selected model and generate a response. The send_event variable captures this event so we can cancel it if needed. On Line 35, we wire up the stop button to cancel ongoing generation, giving users control over long-running requests.

Step 6: Launch the Qwen3 Gradio Chatbot Application Locally

Finally, we create the interface instance and launch the web application.

# Create the interface
interface = create_chatbot_interface(qwen3_instruct, qwen3_thinking)
print("✅ Gradio interface created!")

print("Launching Gradio interface...")
print("Access the chatbot at: http://localhost:7860")
print("Press the 'Stop' button in the output to stop the server\n")

interface.launch()

On Lines 2 and 3, we instantiate our chatbot interface by calling create_chatbot_interface with both model pipelines and confirm successful creation. On Lines 5-7, we print informative messages about how to access the application.

On Line 9, we launch the Gradio interface, which starts a local web server on port 7860. The interface automatically opens in a new browser tab (or provides a link if running in certain environments). Users can now interact with both Qwen3-Instruct and Qwen3-Thinking models through an intuitive chat interface, seamlessly switching between fast responses and deep reasoning as needed.

Sample Qwen3 Chatbot Conversations

To demonstrate how our chatbot blends instructional clarity with deep reasoning, we have included three example interactions below. These showcase how the model responds in both normal and “Thinking Mode,” which users can toggle to reveal step-by-step logic and internal traceability.

In Figure 4, the chatbot responds to a basic science question: “Explain photosynthesis in simple terms.” What is notable here is the clarity and structure of the explanation. The model breaks the process into 4 intuitive steps: sunlight absorption, carbon dioxide intake, water uptake, and glucose production. It then concludes with a simplified chemical equation and adds a metaphor (“a natural kitchen”) to make the concept more relatable. This reflects the model’s instructional capability: concise, clear, and beginner-friendly.

Figure 5 shows the chatbot solving a math problem in Thinking Mode. Here, the chatbot first reveals its internal reasoning trace: a step-by-step breakdown of how it arrived at the answer. It recalls the formula, substitutes values, performs intermediate calculations, and even reflects on whether to include units. This traceability is key: it shows how the model thinks, not just what it knows. For users who want transparency, learning support, or deeper trust, this mode is invaluable.

Finally, in Figure 6, the chatbot provides the correct formula and numerical result, but also distinguishes between the exact answer (49π) and the approximate value (153.94). This demonstrates its ability to handle both symbolic and numeric reasoning.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: July 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

Throughout this lesson, we have explored the groundbreaking Qwen3 model series and its unique dual-mode architecture that revolutionizes how we interact with large language models. We began by understanding the comprehensive model family, from the compact 0.6B parameter variant to the flagship 235B parameter MoE model, each designed to meet different computational requirements and use cases. The architectural innovations (including QK-Norm for training stability, fine-grained expert segmentation in MoE models, and support for 119 languages) demonstrate Qwen3’s position at the forefront of open-source language models.

The 3-stage training process, encompassing general knowledge acquisition, reasoning enhancement, and long-context capability development, produces models that excel across diverse benchmarks while maintaining efficiency through sparse activation patterns.

We then translated this theoretical knowledge into practical implementation by building a functional deep-thinking chatbot. Through hands-on coding, we learned how to set up the development environment, initialize both Instruct and Thinking model pipelines, and create an interactive web interface using Gradio. The implementation showcased key differences between the two modes: Instruct provides fast, direct responses ideal for straightforward queries, while Thinking reveals its step-by-step reasoning process, making it invaluable for complex problem-solving tasks.

We implemented features like conversation history management, dynamic model switching, and collapsible thinking traces, demonstrating best practices for building user-friendly AI applications. The modular code structure we developed provides a solid foundation that can be extended with additional features such as system prompts, temperature controls, or multi-user support.

This lesson serves as the cornerstone of our series on building a comprehensive, safe, and multimodal AI assistant. In the upcoming lessons, we will expand on this foundation by integrating Qwen3-VL for visual understanding (Lesson 2), adding Qwen3-Guard for safety moderation (Lesson 3), and incorporating Qwen3-Embedding for retrieval-augmented generation (Lesson 4). Each addition will transform our basic chatbot into an increasingly sophisticated system capable of understanding images, ensuring safe interactions, and retrieving relevant information from knowledge bases.

The skills and patterns established in this lesson (from pipeline initialization to interface design) will be reused and extended throughout the series. By the end of the complete series, we will have built a production-ready multimodal AI assistant that demonstrates the full potential of the Qwen3 ecosystem. This assistant will combine state-of-the-art language understanding, visual processing, safety guarantees, and knowledge retrieval in a single cohesive application that showcases the power of modern open-source AI.

Citation Information

Mangla, P. “Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/o0x9h

@incollection{Mangla_2026_building-intelligent-chatbot-qwen3-instruct-thinking-models,
  author = {Puneet Mangla},
  title = {{Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/o0x9h},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models appeared first on PyImageSearch.

Google DeepMind’s Gemma 4: MoE, Efficiency Tricks, and Benchmarks

Google DeepMind’s Gemma 4 is one of the most compelling open-weight model releases in recent memory. It’s not just one model; it is a carefully designed family spanning from tiny on-device variants to a 31-billion-parameter powerhouse, all built with multimodal reasoning, long context, and real deployment constraints in mind. And crucially, these models are released under an Apache 2.0 license, meaning you can use, modify, and deploy them commercially without restriction.

In this post, we will peel back the hood and explain what makes Gemma 4 tick, including the architecture, the clever efficiency tricks, the multimodal capabilities, what hardware you actually need to run these models, and how to get started in code. No prior deep knowledge of transformers required, though some familiarity will help.

Whether you are evaluating Gemma 4 for a production use case, curious about the architecture, or just want to know which variant to reach for, this post has you covered.

This lesson is the 1st in a 5-part series on Google DeepMind’s Gemma 4:

1. Google DeepMind’s Gemma 4: MoE, Efficiency Tricks, and Benchmarks (this tutorial)
2. Lesson 2
3. Lesson 3
4. Lesson 4
5. Lesson 5

To learn how Gemma 4’s architecture, Mixture-of-Experts design, multimodal capabilities, and efficiency optimizations work, just keep reading.

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Gemma 4 Model Family Overview: E2B, E4B, 31B, and MoE 26B A4B

Before diving into how these models work, let us first look at the lineup. There are 4 models:

Gemma 4 E2B and E4B: The smallest models in the family, designed to run efficiently on-device (think: your phone). The “E” stands for effective parameters, a concept we’ll unpack below. They support text, images, and even audio.

Gemma 4 31B: A dense 31-billion parameter model. Dense means every parameter participates in every inference pass. Think of it as the “traditional” heavyweight.

Gemma 4 26B A4B: A Mixture-of-Experts model with 26 billion total parameters, but only 4 billion “active” during any given computation (inference). The “A” stands for active parameters. It runs with the speed of a 4B model despite its much larger knowledge capacity.

The lineup spans from phone-friendly to server-grade, so you can pick the right model for your constraints. All 4 models are multimodal; they can reason over images alongside text. The 2 smaller models (E2B and E4B) go a step further and also handle audio.

Every model ships in both a base (pre-trained) and instruction-tuned (IT) version. The instruction-tuned versions are what most practitioners will want to use for tasks like chat, reasoning, and function-calling.

All 4 models are available on Hugging Face, Kaggle, Ollama, LM Studio, and Docker. Also, it can run via Transformers, llama.cpp, MLX, and several other popular inference stacks.

Gemma 4 Capabilities: Reasoning, Multimodal AI, and Thinking Mode

Before getting into architecture, it is worth understanding the capabilities these models were trained and evaluated for. The design choices only make sense in that context.

Gemma 4 Thinking Mode: Chain-of-Thought Reasoning Explained

All Gemma 4 models are designed as capable reasoners with configurable “thinking mode.” When enabled, the model produces an internal chain-of-thought before arriving at its final answer, similar in spirit to what you would see with OpenAI’s o-series or Anthropic’s extended thinking. This is particularly valuable for math, logic, and multi-step planning tasks.

Thinking can be toggled per-request. In the Transformers API, you enable it by passing enable_thinking=True to the apply_chat_template call:

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    enable_thinking=True,  # activates chain-of-thought mode
).to(model.device)

Image Understanding: Object Detection, OCR, and GUI Navigation

The vision capabilities in Gemma 4 are genuinely impressive, especially for an open-weight model. All 4 model sizes could reliably perform bounding-box detection, returning results natively as structured JSON without any special grammar constraints or prompting tricks.

For example, given a UI screenshot and the prompt “What’s the bounding box for the ‘submit’ button?”, the model returns something like:

[{"box_2d": [171, 75, 245, 308], "label": "view recipe element"}]

The coordinates are normalized to a 1000×1000 grid regardless of the original image dimensions, which makes post-processing straightforward. This makes Gemma 4 a strong candidate for tasks like automated UI testing, document parsing, and robotic process automation.

Image captioning was tested across all 4 sizes and all performed well, accurately capturing details such as the type of bird, the architectural style of background buildings, and whether the scene was indoors or outdoors. Even the tiny E2B model produced detailed and accurate captions.

Gemma 4 Code Generation from Images: UI Reconstruction and Vision-to-Code

One standout test: When given each model a screenshot of a webpage and asked it to write the HTML to recreate it. With thinking mode enabled and a token budget of 4,000 output tokens, the larger models (26B A4B and 31B) produced near-faithful reproductions. The smaller E4B model held its own remarkably well, while E2B showed the expected drop-off in fidelity.

This capability to understand a visual layout and translate it into working code has real applications for prototyping, design-to-code workflows, and accessibility tooling.

Gemma 4 Video Understanding: Multimodal Temporal Reasoning

Gemma 4 can process video input, though capabilities differ by size. The smaller E2B and E4B models accept video with audio, treating it as a combined audio-visual signal. The larger 31B and 26B A4B models accept video without audio because they lack an audio encoder, which we will discuss below.

In informal testing with a live concert video, E4B correctly identified the genre of music, the mood of the song lyrics, and the stage setup and crowd. The 31B model gave a detailed description of the visual elements and even identified a brand visible on a large screen, despite not having access to audio. Neither model had been explicitly fine-tuned on video data; this capability emerged from the multimodal training.

Gemma 4 Audio AI: Speech Recognition, Translation, and Audio Q&A

The E2B and E4B models include a dedicated audio encoder, enabling end-to-end speech understanding. This is novel for an open-weight model at this scale. Practically, it means you can send raw audio (as an MP4 or audio file) and ask the model questions about the audio, with no separate transcription step required.

This is particularly useful for:

• Automatic speech recognition (ASR) in a single-model pipeline
• Multilingual audio translation
• Video Q&A where both the speech and visuals matter

Gemma 4 Function Calling: Tool Use and Agentic AI Workflows

Gemma 4 has built-in support for structured function/tool calling, both in text-only and multimodal contexts. This is essential for building agents: systems in which the model needs to decide which tool to invoke, with what arguments, in response to a user request. The fact that this is natively supported (rather than requiring prompt-engineering workarounds) makes Gemma 4 a serious option for agentic workflows running locally or in constrained environments.

Gemma 4 System Prompts: Instruction Control and Chat Behavior

Gemma 4 introduces first-class support for the system role in conversations. In prior Gemma versions, system-level instructions had to be blended into the user turn in ad hoc ways. Now the model is trained to recognize and respect a proper system prompt, which makes deploying it inside structured applications (where you want to set tone, persona, or capabilities) significantly cleaner.

Gemma 4 Architecture Overview: Shared Transformer Design Principles

Despite their size differences, all Gemma 4 models share the same core architectural DNA. Let us go through each shared component one by one.

Gemma 4 Attention Mechanism: Local + Global Interleaved Attention Explained

To appreciate what Gemma 4 does here, you first need to understand what “attention” means in a transformer model.

The classic attention problem: In a standard transformer, every word in your input looks at every other word to figure out context. This is called full or global attention. It is powerful but brutally expensive because the computation grows with the square of the input length. Double your input length, and you quadruple the cost.

Sliding window attention (local attention): Imagine reading a book, but instead of remembering every page you’ve ever read, you can only reference the last 5 pages. That’s sliding window attention. Each token only attends to the N most recent tokens (a “window”), not the entire sequence. This is dramatically cheaper to compute.

Here is the tradeoff made tangible: say you are generating a response to a long legal document. With a sliding window of 512 tokens, any given token looks only at the 512 tokens before it, rather than the entire 10,000-token document. That saves enormous compute, but risks losing context from early in the document.

The interleaving solution: Gemma 4 does not pick one strategy; it alternates between them across layers. Most layers use the efficient sliding window, but every few layers, a full global attention layer kicks in and “resets” the context by attending to everything. Think of it like a student who mostly skims through dense reading, but every few chapters pauses to re-read everything they have covered.

In practice, the E2B model uses a 4-local-to-1-global pattern. All other models use a 5:1 ratio. Crucially, Gemma 4 ensures the final layer is always a global attention layer, so the model’s last word on any sequence is fully informed, a deliberate fix from Gemma 3 where the last layer could end up being local.

Gemma 4 Efficiency Tricks: GQA, K=V Caching, and Memory Optimization

Even with interleaving, global attention layers are still the most expensive part. Gemma 4 layers on three additional tricks to tame the cost.

Grouped Query Attention (GQA)

In standard multi-head attention, every “head” maintains its own set of Key and Value matrices. This creates a large memory footprint because all of these have to be cached during generation (this is called the KV-cache).

GQA is the idea that multiple Query heads can share the same set of Keys and Values. Imagine 8 students all reading from the same textbook instead of each having their own, with the same knowledge and much less paper.

In Gemma 4’s global attention layers, 8 Query heads share a single KV pair. This dramatically reduces what needs to be stored in the cache, which is especially significant because global attention has to cache the entire context (versus the local attention layers, which only cache a small window).

To compensate for any quality loss from fewer KV heads, Gemma 4 doubles the dimensionality of the Keys, giving each shared Key more expressive capacity.

Keys Equal Values (K=V)

Here’s an even bolder efficiency move: in global attention layers, Gemma 4 sets the Key and Value matrices to be identical. Instead of storing both K and V separately in cache, you only need to store one. The KV-cache effectively becomes a K-cache for those layers, cutting memory requirements in half at that level.

This sounds like it might hurt quality significantly, but in practice the performance impact turns out to be modest, a good trade for the memory savings.

p-RoPE: Smarter Positional Encoding

To understand this trick, you need to know how transformers track word order. Because attention has no built-in sense of sequence (unlike an RNN), position is injected into embeddings explicitly. The popular method for this is Rotary Positional Encoding (RoPE).

How RoPE works: Each embedding vector is split into pairs of values. Each pair is thought of as a 2D vector pointing in some direction. RoPE rotates each pair by a position-dependent angle, so earlier words get one rotation, later words get another. By comparing how much two vectors have been rotated, the model can infer their relative distance.

The rotation speeds vary: the first pairs rotate quickly (high frequency) and the last pairs rotate very slowly (low frequency). The high-frequency pairs are great for tracking where a word is. The low-frequency pairs rotate so little that they barely carry positional information at all, making them closer to the raw semantic meaning of the word.

Here is the problem Gemma 4 solves: over very long sequences, even those tiny low-frequency rotations accumulate and start to introduce misleading positional noise into what should be a semantic signal. Think of it like a clock’s hour hand being used to measure seconds, where the movement is technically there but too small to be meaningful and can cause errors.

p-RoPE (pruned RoPE) solves this elegantly: apply rotational encoding only to the first p fraction of pairs, and zero out the rest. If p = 0.25, only the top 25% of pairs (the high-frequency, positional ones) get rotation. The low-frequency pairs are left clean, with pure semantic content and no positional noise. This is especially important in global attention, where the context can span tens of thousands of tokens.

Gemma 4 Vision Encoder: ViT-Based Image Processing Architecture

All four Gemma 4 models are multimodal, meaning they can reason about images as well as text. To make this work, images need to be converted into a format the language model can process. The component responsible for this is the Vision Encoder, built on a Vision Transformer (ViT).

The core idea of a ViT: Rather than treating an image as a grid of pixels, a ViT slices the image into fixed-size patches (typically 16×16 pixels each) and treats each patch like a “word token.” The sequence of patches goes through a transformer, which produces an embedding for each patch capturing its visual content and context.

Handling Variable Aspect Ratios with 2D RoPE

Standard ViTs assume a square input image with a fixed grid of patches. But real-world images come in all shapes (e.g., wide panoramas, tall portraits, and square thumbnails). Forcing every image into a square distorts content and destroys spatial relationships.

Gemma 4 addresses this by using 2D RoPE for its vision encoder. Instead of encoding patches with a single 1D position (patch 1, patch 2, patch 3, etc.), each patch is given a 2D position: its (row, column) coordinates in the image grid. The patch embedding is split into two halves where one half encodes the horizontal position, and the other encodes the vertical position. This way, a patch in the upper-left corner of a wide landscape and a patch in the upper-left corner of a tall portrait both correctly identify themselves as “top-left,” regardless of the total number of patches.

Images are also adaptively resized to maintain the original aspect ratio while ensuring the dimensions are multiples of 16 (the patch size), with padding added where needed.

Soft Token Budget: Controlling Variable Resolution

More patches mean more tokens fed into the language model, which increases computational cost. To give developers control over this, Gemma 4 introduces a soft token budget: a configurable cap on how many visual tokens are processed by the LLM.

Here’s a concrete example. Suppose you set a budget of 280 tokens. The model will resize your image so that the total resulting patches, after pooling every 3×3 patch block into a single embedding, stays within 280. A budget of 1120 tokens lets high-resolution images through with much more visual detail; a budget of 70 tokens dramatically downsamples the image. The right budget depends on your task:

• Describing a photo? 70–140 tokens is probably fine.
• Reading a scanned invoice with fine print? You’d want 560–1120 tokens.
• Analyzing consecutive video frames quickly? Lower budgets keep things fast.

Linear Projection: Bridging Vision and Language

The patch embeddings produced by the ViT live in a different dimensional space than the word embeddings Gemma 4 was trained on. Feeding mismatched embeddings into the language model would be like asking someone to add meters and kilograms, which makes no sense.

To solve this, a small neural network called a linear projection learns to map vision embeddings into the exact dimensional space Gemma 4 expects. This projection is trained alongside the language model so it perfectly aligns the two embedding spaces. A normalization step (RMSNorm) follows the projection to ensure the scale of visual embeddings matches what the transformer layers anticipate.

Gemma 4 Architecture Variants: Dense vs MoE vs On-Device Models

Now that you understand what all Gemma 4 models share, let us look at what makes each variant distinctive.

Gemma 4 31B: The Dense Baseline

The 31B model is the most architecturally conventional in the family. It is a dense transformer, meaning every parameter is used on every forward pass. Think of it as a large, all-purpose Swiss Army knife: every tool is always there, every tool can always be used.

Its architecture closely follows Gemma 3’s 27B model in spirit, but applies all the global attention improvements we’ve described: K=V, 8-query GQA, doubled Key dimensions, and p-RoPE. It has 60 layers (slightly fewer than Gemma 3’s 27B model with 62 layers) but compensates with a wider hidden dimension, meaning more parameters per layer rather than more layers.

For most inference scenarios that require a powerful, capable model without the complexity of MoE routing, this is the model to reach for.

Gemma 4 26B A4B MoE: Sparse Experts and Efficient Inference Explained

This is where things get architecturally interesting. The 26B A4B model uses a design called Mixture of Experts (MoE) to achieve something remarkable: the knowledge capacity of a 26-billion-parameter model at roughly the inference cost of a 4-billion-parameter model.

How Mixture of Experts Works

In a standard (dense) transformer, every layer contains a single large feedforward neural network (FFNN) that processes every token. In a MoE layer, that single FFNN is replaced by a collection of smaller FFNNs called experts, plus a lightweight router network.

When a token arrives at a MoE layer, here’s what happens step by step:

• The router examines the token’s embedding and assigns a probability score to each expert.
• The top-scoring experts are selected (in Gemma 4, 8 out of 128 experts are chosen).
• Each selected expert processes the token independently and produces an output.
• The outputs are weighted by the router’s probability scores and summed together.

This means for any given token, only 8 experts are doing work, while the other 120 are idle. The total number of parameters that get loaded into memory (the “sparse” parameters) is 26B. But the number doing active computation (the “active” parameters) is only 4B. Hence: 26B A4B.

A good analogy: imagine a hospital with 128 specialist doctors, but any given patient only sees 8 of them during their visit. The hospital has the collective knowledge of all 128 doctors, but each consultation only draws on a relevant subset.

The Shared Expert

Gemma 4’s MoE adds one more element: a shared expert that is always activated for every single token, regardless of what the router decides. This expert is three times larger than the other experts.

The intuition is compelling. Some knowledge is universally useful (e.g., grammar, common-sense reasoning, and factual recall) and should always be applied. The shared expert holds this general knowledge. The routed experts hold more specialized knowledge that is selectively engaged depending on the content. This is similar to how you would always use your native language’s grammar rules (shared expert), but only pull out domain-specific vocabulary when discussing, say, molecular biology (a selected expert).

Gemma 4 E2B and E4B: On-Device Multimodal AI Models for Edge Deployment

These are the smallest and most novel models in the family. They are designed to run on devices with severely limited RAM, with smartphones being the primary target. Two key innovations enable this: Per-Layer Embeddings and an Audio Encoder.

Per-Layer Embeddings (PLE): Teaching Each Layer Its Own Vocabulary

In a standard transformer, each token is looked up in a single embedding table at the very start. That means one embedding per token, used everywhere. A richer context comes from stacking many transformer layers on top.

Per-Layer Embeddings take a different approach. Each token has not one embedding, but a separate embedding for every layer in the model. Continuing our analogy: instead of greeting a visitor with one name badge, you give them a different badge for each room they will enter, with each badge describing their role in the context of that room’s purpose.

For the E2B model, this means 262,144 vocabulary tokens × 35 layers × 256 dimensions per layer-embedding. That’s a large table, but here’s the key insight: this table lives in flash storage (like your phone’s SSD), not in RAM. RAM is precious and fast; flash is abundant and cheap. During inference, the needed embeddings are fetched from flash memory once at the start, then used at each layer.

At each layer, a gating function decides how to weight the values in the fetched embedding, effectively letting the model emphasize different aspects of a token’s meaning at different depths. The resulting embedding is projected up to the full model dimension and added into the main processing stream, functioning as a kind of continuous “reminder” to each layer of what the original token meant, preventing that meaning from getting diluted as context accumulates.

The “E” in E2B means effective parameters, referring to the parameters that actually reside in RAM and do computation. The large layer-embedding table is intentionally excluded from this count because it sits in flash, not in working memory.

The Audio Encoder

The E2B and E4B models go one step further: they accept raw audio as input, enabling tasks like speech recognition, audio translation, and voice-based Q&A.

Audio processing follows a three-stage pipeline before the language model ever sees it:

Stage 1. Feature Extraction: The raw audio waveform is converted into a mel-spectrogram, which is a 2D image-like representation where the horizontal axis represents time and the vertical axis represents frequency. This is similar to how sheet music represents music: time flows left to right, and the vertical position tells you the pitch. The mel scale emphasizes frequency ranges the human ear is most sensitive to.

Stage 2. Chunking: The mel-spectrogram is divided into overlapping chunks, turning the continuous audio signal into a structured sequence of frames ready for processing.

Stage 3. Downsampling with Convolutions: Two 2D convolutional layers process and compress these chunks, reducing the sequence length into a manageable number of “soft tokens” (continuous, dense embeddings rather than discrete word tokens). This is the audio equivalent of the ViT’s patch pooling: it reduces a large number of raw signals into a compact, information-rich sequence.

The resulting audio embeddings pass through a Conformer encoder, a transformer-style architecture augmented with convolutional modules, which is well-suited for sequential signal data such as audio. The Conformer’s output is then linearly projected into Gemma 4’s embedding space, exactly as we saw with the vision encoder.

The beauty of this design is that it’s modality-agnostic in spirit: whether it’s a word, an image patch, or an audio chunk, the final product is always a sequence of aligned embeddings that the language model can reason over uniformly.

Gemma 4 Hardware Requirements: GPU VRAM and Inference Cost Breakdown

Understanding memory requirements is critical before committing to a deployment setup. Here are the approximate GPU or TPU memory requirements for running inference at different precision levels.

At full 16-bit precision, the 31B model needs roughly 60 GB of VRAM, which is equivalent to two A100 80GB GPUs or a single H100. But at 4-bit quantization, the same model fits in about 17 GB, which means a single RTX 4090 or A10G becomes viable.

The 26B A4B model is interesting: its full-precision footprint of 48 GB looks large, but because only 4B parameters are active during inference, it runs significantly faster than the 31B despite needing less memory. At 4-bit, it drops to 15.6 GB.

The E2B and E4B models, at 4-bit quantization, fit in 3–5 GB of VRAM, placing them in genuinely on-device territory for modern phones and edge hardware. The E suffix models are especially designed for this: their PLE (Per-Layer Embeddings) tables live in flash storage, so the actual RAM footprint is even smaller than these numbers suggest during full inference runs on mobile devices.

Gemma 4 Benchmarks: LMArena Elo Scores and Multimodal Performance Results

Gemma 4’s large models set a new bar for what’s achievable in the open-weight space at this parameter count.

The 31B dense model achieves an estimated LMArena Elo score of 1,452 on text-only evaluations, placing it competitively with models that are significantly larger. The 26B A4B MoE model reaches 1,441, which is remarkable given that it uses only 4 billion active parameters. To put that in context: these scores are competitive with several closed-source models from mid-2024.

Multimodal performance follows a similar pattern. Even the vision and audio capabilities were comparable in quality to the text performance, and not degraded by the multimodal conditioning. All model sizes demonstrated strong OCR, object detection, scene description, and audio understanding.

On coding and agentic benchmarks, Gemma 4 shows notable improvements over Gemma 3, partly due to the expanded context window (128K for small models, 256K for large ones), the native function-calling support, and the thinking-mode capability.

How to Run Gemma 4: Transformers, llama.cpp, MLX, and Cloud Deployment Options

Google and the community have built Gemma 4 support into virtually every major inference stack. Here’s a quick summary to help you choose.

• Hugging Face Transformers: The most fully featured option for Python users. It supports all modalities, thinking mode, function calling, and the full Processor API for handling mixed text/image/audio inputs. It is the best choice for research, fine-tuning, and flexible experimentation.
• Llama.cpp: Offers highly optimized CPU and GPU inference, particularly valuable if you’re running on Apple Silicon or hardware without NVIDIA GPUs. Gemma 4 is supported in recent builds, with GGUF quantization enabling the small models to run on consumer hardware.
• MLX: The framework of choice for Apple Silicon, offering native Metal GPU acceleration. The E2B and E4B models run surprisingly fast on M-series chips via MLX, making on-Mac deployment practical.
• transformers.js: Enables in-browser inference via WebGPU. Gemma 4’s small models can run directly in a web browser (no server required), which opens up genuinely private, fully offline applications.
• Mistral.rs: A Rust-based inference engine with strong performance characteristics for production deployments.

For cloud production environments, Gemma 4 is available via the Gemini API, Google Cloud’s Vertex AI, Cloud Run, and GKE with GPU nodes. The Gemini API option is the lowest-friction path for managed serving without infrastructure work.

Fine-Tuning Gemma 4: LoRA, QLoRA, and TRL Training Pipeline Guide

One interesting observation from the Hugging Face team: Gemma 4 was difficult to demonstrate through fine-tuning examples because the base instruction-tuned models are already so capable. That said, fine-tuning is well-supported for domain specialization, style adaptation, or building task-specific versions.

TRL (Transformer Reinforcement Learning): The primary recommended library for supervised fine-tuning. It supports QLoRA (quantized LoRA), which dramatically reduces the memory requirements for fine-tuning, making it possible to fine-tune the 31B model on a machine with two consumer-grade GPUs if combined with 4-bit quantization. Fine-tuning is also supported on Vertex AI via TRL if you’d prefer a managed training environment.

Unsloth Studio: A no-code fine-tuning interface for users who want to adapt Gemma 4 without writing training code. It supports Gemma 4 with memory optimizations baked in.

For a full fine-tuning pipeline in code, the key is using QLoRA via Hugging Face’s peft and trl libraries, targeting the attention and feedforward projection layers. Google also provides official guides for LoRA fine-tuning via Keras, PyTorch, and the Gemma library itself.

Gemma 4 Prompt Formatting: Chat Templates and Multimodal Input Structure

Gemma 4 follows a specific chat template that you should be aware of when building applications. The instruction-tuned models expect input in a structured multi-turn format. When using Hugging Face Transformers, always use processor.apply_chat_template() rather than constructing prompts manually. This ensures special tokens are correctly inserted and the model receives input in the format it was trained on.

For multimodal inputs, images and audio are passed as dictionary entries alongside text in the message content list:

messages = [
    {
        "role": "user",
        "content": [
            # For image input:
            {"type": "image", "url": "https://example.com/image.png"},
            # Or for local audio:
            {"type": "audio", "path": "/path/to/audio.mp3"},
            # Text always accompanies the media:
            {"type": "text", "text": "Describe what you see/hear."},
        ],
    }
]

For video with audio (E2B and E4B only), pass load_audio_from_video=True in the apply_chat_template call. For larger models, omit this flag since they do not have an audio encoder.

Which Gemma 4 Model to Use: E2B vs E4B vs 26B MoE vs 31B Comparison

With 4 variants available, the choice comes down to a few key questions.

If you are building something that runs on a phone or edge device with less than 6–8 GB of RAM available for the model, the E2B or E4B are your options, and they are genuinely capable. E4B is worth the extra memory if you are doing audio-visual tasks. At 4-bit quantization, E2B runs in about 3 GB, which fits on most modern Android and iOS devices.

If you are running on a single GPU in the 16–24 GB range (RTX 3090, 4090, A10G), the 26B A4B at 4-bit quantization (15.6 GB) gives you the best intelligence-per-dollar, running at 4B-speed throughput.

If you need maximum capability and have the hardware for it (2× A100 or H100), the 31B dense model at BF16 or the 26B A4B at 16-bit precision are both strong choices. The 31B is architecturally simpler; the 26B A4B provides better throughput if you’re processing high request volumes.

If you are doing audio tasks at all, you must use E2B or E4B, since the larger models do not have an audio encoder.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: July 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

Gemma 4 is best understood not as a single model but as a thoughtfully tiered family, each member engineered for a specific place in the hardware spectrum, from a smartphone to a data center GPU cluster.

The two small models (E2B and E4B) push the frontier of what is possible on-device by storing large embedding tables in flash memory rather than RAM, and by packing audio understanding alongside vision and text in a package that fits in just a few gigabytes.

The 26B A4B MoE model achieves something that still feels almost counterintuitive: the knowledge depth of a 26-billion-parameter model running at roughly the speed and cost of a 4-billion-parameter model, thanks to sparse expert routing.

The 31B dense model serves as the reliable, architecturally simple heavyweight for applications that need maximum capability without the added complexity of MoE.

Across all variants, Gemma 4 shares a core set of architectural decisions that compound in value: interleaved local-and-global attention tames the cost of long contexts; grouped query attention and the K=V cache trick shrink the memory footprint of those global layers; and pruned positional encoding keeps semantic meaning clean even across hundreds of thousands of tokens.

These are not isolated optimizations; they are a coherent strategy for squeezing frontier-level intelligence into constrained environments.

On the capability side, what sets Gemma 4 apart from prior open-weight releases is the breadth of what works out of the box. Native structured output for object detection, code generation from screenshots, audio Q&A, configurable thinking mode, and function-calling support all come without special prompting tricks or external scaffolding.

The Apache 2.0 license is a major advantage for commercial use, allowing you to deploy, modify, and build on these models without restriction.

If you take one thing away from this post, let it be this: the right way to approach Gemma 4 is not to ask “which is the best model?” but rather “what are my actual constraints — memory, latency, modality, hardware — and which variant is engineered for exactly that?”

The answer is almost certainly one of these four. The rest of this series will help you put whichever one you choose to work.

Citation Information

Thakur, P. “Google DeepMind’s Gemma 4: MoE, Efficiency Tricks, and Benchmarks,” PyImageSearch, S. Huot and A. Sharma, eds., 2026, https://pyimg.co/uqxzw

@incollection{Thakur_2026_google-deepminds-gemma-4-moe-efficiency-tricks-benchmarks,
  author = {Piyush Thakur},
  title = {{Google DeepMind's Gemma 4: MoE, Efficiency Tricks, and Benchmarks}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma},
  year = {2026},
  url = {https://pyimg.co/uqxzw},
}

Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to join the PyImageSearch Newsletter and download my FREE 17-page Resource Guide PDF on Computer Vision, OpenCV, and Deep Learning.

Join the Newsletter!

Website

The post Google DeepMind’s Gemma 4: MoE, Efficiency Tricks, and Benchmarks appeared first on PyImageSearch.

Run an Apache Airflow DAG with Docker Compose and PostgreSQL

In Lesson 1, we designed a production-grade document ingestion pipeline using FastAPI, Apache Airflow, and PostgreSQL. We defined the data model, built an idempotent 5-task DAG, structured our project for separation of concerns, and established the architectural principles that make the system reliable.

Now, in Lesson 2, we bring that architecture to life.

In this lesson, you will operationalize the ingestion pipeline by running it inside Docker, uploading real documents, triggering DAG runs, inspecting logs, and validating results directly in PostgreSQL. You will also explore failure scenarios, observability patterns, and the practical limits of using Airflow for ML-style workloads.

By the end of this lesson, you will not only understand how the pipeline works, but how to run it, debug it, monitor it, and reason about its behavior in production.

This lesson is the last in a 2-part series on building production-grade document ingestion pipelines for RAG systems:

1. Apache Airflow Document Ingestion Pipeline for RAG Systems
2. Run an Apache Airflow DAG with Docker Compose and PostgreSQL (this tutorial)

To learn how to deploy, execute, monitor, and validate a production-ready ingestion pipeline with Apache Airflow, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Project Structure

In Lesson 1, we focused on architectural boundaries. Now we look at the same structure through a runtime lens: what runs where, and what each container actually executes.

├── airflow_project/        # Orchestration + runtime layer
│   ├── dags/              # Airflow DAG definitions
│   │   └── ingest_documents_dag.py  # Main DAG: PDF → chunks pipeline
│   │
│   ├── ingestion_service/ # FastAPI REST API for file uploads
│   │   ├── __init__.py
│   │   ├── main.py      # Upload endpoint with deduplication
│   │   └── requirements.txt  # FastAPI + Uvicorn dependencies
│   │
│   ├── docker-compose.yml  # Defines Postgres, Airflow, API services
│   ├── Dockerfile               # Airflow container image
│   ├── Dockerfile.service   # FastAPI service container image
│   └── init-db.sh              # PostgreSQL database initialization script
│
├── shared/      # Reusable business logic (no Airflow deps)
│   ├── data_models/ # Pydantic schemas (API validation layer)
│   │   ├── __init__.py
│   │   └── models.py  # Document, Chunk, PipelineRun schemas
│   │
│   ├── parsing/                # Document processing logic
│   │   ├── __init__.py
│   │   ├── pdf_parser.py # PyPDF text extraction (page-level)
│   │   ├── chunker.py    # Sliding window text chunking
│   │   └── deduplication.py # Content hash comparison utilities
│   │
│   ├── storage/            # Database interaction layer
│   │   ├── __init__.py
│   │   ├── database.py     # SQLAlchemy session management
│   │   └── models.py       # ORM models (tables definition)
│   │
│   ├── utils/               # Low-level helpers
│   │   ├── __init__.py
│   │   ├── hashing.py      # SHA-256 file & text hashing
│   │   └── logging.py      # Structured logging utilities
│   │
│   ├── __init__.py
│   └── requirements.txt  # Shared dependencies (SQLAlchemy, PyPDF, etc.)

When you run:

docker compose up --build

This structure turns into 5 running services.

What Runs Inside Each Container

Airflow Webserver and Scheduler

• Load DAGs from dags/
• Import parsing and storage logic from shared/
• Connect to PostgreSQL
• Read uploaded files from the shared volume

Ingestion Service

• Runs main.py
• Accepts PDF uploads
• Writes files into /tmp/ml_orchestration/uploads
• Inserts rows into PostgreSQL

PostgreSQL

• Stores Airflow metadata
• Stores documents, chunks, and pipeline_runs

Why shared/ Is Mounted Everywhere

Both the Airflow containers and the ingestion service mount the shared/ directory.

This guarantees:

• The API and DAG use identical parsing logic
• Hashing behaves consistently
• Database models stay synchronized

No duplicated code.

No version drift.

The Shared Upload Volume

The /tmp/ml_orchestration/uploads volume is mounted into both:

• ingestion-service
• airflow containers

This is critical.

The API writes files.

Airflow reads the same files directly from disk.

There is no file transfer, no API polling, and no network hop.

This design eliminates an entire class of distributed file consistency problems.

Why This Matters in Practice

At runtime:

• The API handles fast, user-facing work.
• Airflow handles scheduled, asynchronous work.
• PostgreSQL persists state.
• The shared module ensures consistency.
• Docker isolates everything into reproducible containers.

Lesson 1 taught you how the system is designed.

Lesson 2 shows you how that design behaves when executed.

PDF Parsing and Text Chunking Logic for Airflow DAG

The DAG calls functions like parse_pdf() and chunk_text(), but these are not defined in the DAG file. They live in shared/parsing/ where both Airflow and future Argo Workflows can use them. Figure 1 shows the chunking pipeline.

Here is the PDF parsing code from shared/parsing/pdf_parser.py:

def parse_pdf(file_path: str) -> List[dict]:
    """
    Parse a PDF file and extract text page by page.
    
    Args:
        file_path: Path to PDF file
        
    Returns:
        List of dictionaries with page_number and text
    """
    logger.info(f"Parsing PDF: {file_path}")
    
    if not os.path.exists(file_path):
        raise FileNotFoundError(f"PDF file not found: {file_path}")
    
    try:
        reader = PdfReader(file_path)
        pages = []
        
        for i, page in enumerate(reader.pages):
            text = page.extract_text()
            metadata = reader.metadata or {}
            
            pages.append({
                'page_number': i + 1,
                'text': text,
                'metadata': {
                    'title': metadata.get('/Title', ''),
                    'author': metadata.get('/Author', ''),
                    'total_pages': len(reader.pages)
                }
            })
        
        logger.info(f"Extracted {len(pages)} pages from PDF")
        return pages
        
    except Exception as e:
        logger.error(f"Failed to parse PDF {file_path}: {str(e)}")
        raise

This function uses PyPDF’s PdfReader to load the PDF. It iterates over each page, calls extract_text(), and bundles the text with page metadata. The result is a list of dictionaries, one per page.

Notice we extract metadata like title and author if available. This can be useful for citations or provenance tracking later. We also include the total page count so downstream tasks know if they have partial data.

The error handling re-raises exceptions after logging. This ensures failures propagate to Airflow where they can trigger retries or mark the document as failed.

Here is the chunking code from shared/parsing/chunker.py:

def chunk_text(
    text: str,
    chunk_size: int = 512,
    overlap: int = 50,
    min_chunk_size: int = 100
) -> List[str]:
    """
    Split text into overlapping chunks.
    
    Args:
        text: Input text to chunk
        chunk_size: Target size in words (approximate token proxy)
        overlap: Number of overlapping words between chunks
        min_chunk_size: Minimum chunk size in words (discard smaller)
        
    Returns:
        List of text chunks
    """
    logger.info(f"Chunking text: {len(text)} chars, chunk_size={chunk_size}, overlap={overlap}")
    
    words = text.split()  # Simple whitespace splitting (production would use proper tokenizers)
    chunks = []
    start = 0
    
    while start < len(words):
        end = min(start + chunk_size, len(words))
        chunk_words = words[start:end]
        chunk = ' '.join(chunk_words)
        
        if len(chunk_words) >= min_chunk_size:
            chunks.append(chunk)
        
        start += (chunk_size - overlap)
    
    logger.info(f"Created {len(chunks)} chunks")
    return chunks

This function uses a sliding window approach — imagine sliding a frame across a long document, capturing a portion of text, then sliding forward while keeping some overlap with the previous frame.

We split the text into words using whitespace (production systems would use proper tokenizers like tiktoken, but whitespace splitting works as a reasonable approximation). We extract a window of chunk_size words, join them back into a string, and add to the chunks list. Then we slide the window forward by chunk_size - overlap words to create the next chunk.

The overlap is critical for retrieval quality. If an important sentence or concept spans a chunk boundary, the overlap ensures it appears in full within at least one chunk. Without overlap, you might split a key phrase across two chunks and miss it during semantic search.

We also have a min_chunk_size parameter. Very small chunks (just a few words) usually lack enough context to be semantically meaningful, so we discard them.

Here is the content hashing code from shared/utils/hashing.py:

def hash_content(content: str) -> str:
    """
    Compute SHA-256 hash of text content.
    
    Args:
        content: Text to hash
        
    Returns:
        Hexadecimal hash string
    """
    return hashlib.sha256(content.encode('utf-8')).hexdigest()


def hash_file(file_path: str) -> str:
    """
    Compute SHA-256 hash of file content.
    
    Args:
        file_path: Path to file
        
    Returns:
        Hexadecimal hash string
    """
    hash_obj = hashlib.sha256()
    
    with open(file_path, 'rb') as f:
        for chunk in iter(lambda: f.read(4096), b''):
            hash_obj.update(chunk)
    
    return hash_obj.hexdigest()

The hash_content() function encodes text as UTF-8 bytes and computes its SHA-256 hash. The hash_file() function reads a file in 4KB chunks (to handle large files without loading them entirely into memory) and computes the cumulative hash.

These hashes are what enable deduplication. Two identical chunks will always produce the same hash, so we can check for existence in the database before inserting.

Why Sharing This Logic Matters

By putting parsing and chunking in shared/, we ensure consistency across different orchestration systems. When we introduce Argo Workflows in a future lesson, it will use the exact same parse_pdf() and chunk_text() functions. This means documents processed by Airflow and documents processed by Argo will have identical chunk boundaries, identical hashes, and identical behavior.

This is critical for systems where you might migrate from one orchestrator to another or run hybrid setups.

Now let’s see how this all runs in Docker.

Docker Orchestration and Runtime for Airflow and FastAPI

We have examined the code. Now let’s see how Docker brings it all together. Figure 2 shows the container topology.

Here is the complete docker-compose.yml:

version: '3.8'

x-airflow-common:
  &airflow-common
  image: apache/airflow:2.7.3-python3.11
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'false'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session'
    PYTHONPATH: /opt/airflow/shared
    ML_ORCHESTRATION_DB_URI: postgresql://mlops:mlops_password@postgres:5432/ml_orchestration
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ../shared:/opt/airflow/shared
    - /tmp/ml_orchestration/uploads:/tmp/ml_orchestration/uploads
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    &airflow-common-depends-on
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
      - ./init-db.sh:/docker-entrypoint-initdb.d/init-db.sh
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 10s
      retries: 5
      start_period: 5s
    ports:
      - "5432:5432"

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8974/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: always

  airflow-init:
    <<: *airflow-common
    entrypoint: /bin/bash
    command:
      - -c
      - |
        airflow db init
        airflow users create \
          --username admin \
          --firstname Admin \
          --lastname User \
          --role Admin \
          --email admin@example.com \
          --password admin
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
    user: "0:0"

  ingestion-service:
    build:
      context: .
      dockerfile: Dockerfile.service
    ports:
      - "8000:8000"
    environment:
      ML_ORCHESTRATION_DB_URI: postgresql://mlops:mlops_password@postgres:5432/ml_orchestration
      UPLOAD_DIR: /tmp/ml_orchestration/uploads
    volumes:
      - ./ingestion_service:/app/ingestion_service
      - ../shared:/app/shared
      - /tmp/ml_orchestration/uploads:/tmp/ml_orchestration/uploads
    depends_on:
      postgres:
        condition: service_healthy
    restart: always

volumes:
  postgres-db-volume:
  /tmp/ml_orchestration/uploads:

Let us break down the key parts.

The x-airflow-common anchor (starting with &) defines configuration shared by all Airflow containers. This is a YAML feature that lets us avoid repeating the same settings. The <<: *airflow-common syntax merges that configuration into each service.

Environment Variables

The AIRFLOW__CORE__EXECUTOR: LocalExecutor runs tasks in separate processes on the same machine (unlike SequentialExecutor which runs one task at a time, or CeleryExecutor which distributes across workers). Perfect for development and moderate workloads.

The AIRFLOW__DATABASE__SQL_ALCHEMY_CONN points to the PostgreSQL container using the service name postgres as the hostname — Docker Compose automatically creates a network where services can reach each other by name.

The PYTHONPATH: /opt/airflow/shared adds the shared module to Python’s import path, allowing the DAG to import from shared.storage, shared.parsing, etc. Without this, imports would fail.

The ML_ORCHESTRATION_DB_URI connects to our application database (ml_orchestration), which is separate from Airflow’s metadata database.

Volume Configuration

Volumes mount local directories into containers. The ./dags:/opt/airflow/dags mapping makes your local DAG files immediately visible to Airflow — edit locally, see changes in the container.

The critical volume is /tmp/ml_orchestration/uploads:/tmp/ml_orchestration/uploads, which mounts the same directory into both Airflow and the ingestion service. This shared storage lets Airflow read files the ingestion service writes, eliminating the “file not found” errors you would get with isolated containers.

Service Definitions

The postgres service runs PostgreSQL 15. It uses a volume (postgres-db-volume) for persistent storage. Without this volume, restarting the container would erase all data. The init-db.sh script is mounted into /docker-entrypoint-initdb.d/, which PostgreSQL runs automatically on first startup. This creates the ml_orchestration database and mlops user.

The healthcheck tests if PostgreSQL is ready. Other services wait for this health check to pass before starting.

The airflow-webserver service runs the Airflow UI. It inherits all settings from airflow-common and adds a webserver command. It exposes port 8080, which you access in your browser.

The airflow-scheduler service runs the scheduler, which triggers DAG runs according to the schedule interval. This is the component that wakes up every minute and checks if the DAG should run.

The airflow-init service is a one-time initialization container. It runs airflow db init to create Airflow’s metadata tables and airflow users create to set up the admin user. After this completes, the container exits. You do not interact with it after the first startup.

The ingestion-service service runs the FastAPI app. It builds from Dockerfile.service, which installs dependencies and runs uvicorn. It mounts the ingestion service code and shared module. It also mounts the uploads volume so it can write files.

Building the Ingestion Service Container

The Dockerfile.service defines how the FastAPI container is built. Here is the complete file:

# airflow_project/Dockerfile.service
FROM python:3.11-slim

WORKDIR /app

# Copy shared module first
COPY shared/ /app/shared/
RUN pip install --no-cache-dir -r /app/shared/requirements.txt

# Copy service dependencies
COPY airflow_project/ingestion_service/requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt

# Copy service code
COPY airflow_project/ingestion_service/ /app/ingestion_service/

# Set Python path to include shared module
ENV PYTHONPATH="/app:/app/shared"

EXPOSE 8000

CMD ["uvicorn", "ingestion_service.main:app", "--host", "0.0.0.0", "--port", "8000"]

This Dockerfile uses a multi-layer approach. First, we copy and install the shared module dependencies. This layer is cached and only rebuilt when shared dependencies change. Then we install the service-specific dependencies. Finally, we copy the application code.

The PYTHONPATH environment variable is critical. It tells Python where to find modules. By including both /app and /app/shared, we can import from both ingestion_service.main and shared.storage.models without issues.

The CMD runs uvicorn, the Asynchronous Server Gateway Interface (ASGI) server for FastAPI. The --host 0.0.0.0 makes the server accessible from outside the container (necessary for Docker port mapping). The --port 8000 matches the port we expose in docker-compose.yml.

Database Initialization Script

The init-db.sh script creates the ml_orchestration database and mlops user when PostgreSQL starts for the first time. Here is the complete script:

#!/bin/bash
# airflow_project/init-db.sh
set -e

# Create additional database and user for ML orchestration
# This script is idempotent - safe to run multiple times
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
    -- Create database only if it doesn't exist
    SELECT 'CREATE DATABASE ml_orchestration'
    WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'ml_orchestration')\gexec
    
    -- Create user only if it doesn't exist
    DO
    \$\$
    BEGIN
        IF NOT EXISTS (SELECT FROM pg_catalog.pg_user WHERE usename = 'mlops') THEN
            CREATE USER mlops WITH PASSWORD 'mlops_password';
        END IF;
    END
    \$\$;
    
    -- Grant privileges (idempotent operation)
    GRANT ALL PRIVILEGES ON DATABASE ml_orchestration TO mlops;
    
    -- Connect to ml_orchestration and grant schema privileges
    \c ml_orchestration
    GRANT ALL ON SCHEMA public TO mlops;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO mlops;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO mlops;
EOSQL

echo "ML orchestration database setup completed successfully"

This script demonstrates several important PostgreSQL patterns. The set -e command makes the script exit immediately if any command fails, preventing partial setup.

The SELECT 'CREATE DATABASE ...' WHERE NOT EXISTS ... \gexec pattern creates the database only if it does not already exist. This is idempotent — running the script multiple times produces the same result.

The DO $$ ... END $$ block is a PL/pgSQL anonymous code block that checks if the user exists before creating it. Again, this makes the script safe to rerun.

The GRANT ALL PRIVILEGES statements give the mlops user full access to the ml_orchestration database. The ALTER DEFAULT PRIVILEGES commands ensure that future tables and sequences created by any user are automatically accessible to mlops. This prevents permission errors when Airflow creates tables.

The \c ml_orchestration command switches the connection to the newly created database so we can grant schema-level privileges.

This initialization script runs automatically because docker-compose mounts it into /docker-entrypoint-initdb.d/. PostgreSQL executes all scripts in that directory on first startup.

Why This Structure Works

This setup gives us isolated, reproducible environments. Everything runs in containers. There is no “works on my machine” problem. The same docker-compose.yml runs identically on macOS, Linux, and Windows (with Docker Desktop).

The shared uploads volume is the key to inter-service communication. Both Airflow and the ingestion service see the same files. Airflow does not need to call the ingestion API to get files; it just reads from the shared directory.

The dependency management (depends_on with health checks) ensures services start in the correct order. PostgreSQL must be ready before Airflow tries to connect.

Now let’s see how to run this system from scratch.

Running the Apache Airflow and FastAPI Document Ingestion Pipeline

To start the system, open a terminal, navigate to airflow_project/, and run:

docker compose up --build

The --build flag ensures Docker rebuilds images if you changed any code. Docker will pull images, build containers, and start all services. You will see logs streaming from all 5 containers.

After about 30 seconds, you should see messages such as the following:

airflow-init-1         | Admin user admin created
airflow-webserver-1    | Listening at http://0.0.0.0:8080
airflow-scheduler-1    | Starting the scheduler
ingestion-service-1    | INFO:     Uvicorn running on http://0.0.0.0:8000
postgres-1             | database system is ready to accept connections

This means all services are up and healthy.

Verify the Setup

Open your browser and go to:

• Airflow UI: http://localhost:8080
• Ingestion API Docs: http://localhost:8000/docs

Log into Airflow with username admin and password admin. You should see the DAGs page.

Find the ingest_documents_dag row. Make sure the toggle on the left is enabled (blue or green). This allows the scheduler to run the DAG.

Go to http://localhost:8000/docs. This is the interactive API documentation powered by Swagger. You should see a green POST box labeled /documents.

If all 3 URLs load successfully, your system is running.

Stopping the System

To stop all containers, press Ctrl+C in the terminal where docker compose up is running. Then run:

docker compose down

This stops and removes containers but preserves volumes (your data persists). If you want to completely reset everything (delete all data), run:

docker compose down -v

The -v flag removes volumes, giving you a fresh start.

Now let us upload a document and trigger the pipeline.

Uploading Documents via the API

The ingestion service exposes a single endpoint: POST /documents. You can call it with curl or use the Swagger UI for convenience. Let us use Swagger for this demo.

In the Swagger UI (http://localhost:8000/docs), click the green POST /documents box to expand it. Click "Try it out" in the top right corner. You should see a file upload field.

Click "Choose File" and select any PDF from your computer. For testing, use a small PDF (a few pages). Click "Execute" to send the request.

You should receive a 201 Created response with JSON similar to the following:

{
  "id": 1,
  "filename": "sample.pdf",
  "file_path": "/tmp/ml_orchestration/uploads/1769421678.801241_sample.pdf",
  "content_hash": "a7f3b2c8d9e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7",
  "file_size": 23456,
  "mime_type": "application/pdf",
  "status": "PENDING",
  "created_at": "2026-01-25T10:30:00",
  "updated_at": "2026-01-25T10:30:00"
}

The important fields are id (1 in this case) and status (PENDING). This document is now in the database waiting for Airflow to process it.

If you upload the same file again, you will get a 409 Conflict error:

{
  "detail": "Document already exists with ID 1"
}

This is the deduplication logic working. The system detected the identical content hash and rejected the duplicate.

Using curl

If you prefer the command line, use curl:

curl -X POST "http://localhost:8000/documents" \
  -H "Content-Type: multipart/form-data" \
  -F "file=@/path/to/your/document.pdf"

Replace /path/to/your/document.pdf with the actual path to your PDF.

Now that we have a pending document, let’s see Airflow process it.

Triggering and Monitoring Apache Airflow DAGs

The DAG runs automatically every minute. If you just uploaded a document, wait up to 60 seconds and the scheduler will trigger a run.

Manual Trigger (Optional)

If you do not want to wait, you can manually trigger the DAG. In the Airflow UI, click on ingest_documents_dag to open the DAG detail page.

You should see a graph view with your five tasks connected by arrows. On the top right, click the play button (triangle icon) and select “Trigger DAG”. This starts an immediate run.

Monitoring Execution

The DAG runs page shows all executions. Click on the latest run (top of the list). The status will progress from “running” (yellow) to “success” (green) or “failed” (red).

Each task should turn green as it completes. The entire run takes about 10-30 seconds, depending on your PDF size.

Viewing Logs

Click on any task box (e.g., parse_documents). A panel appears on the right. Click “Log” to see the task output.

You should see log messages similar to the following:

[2026-01-25, 10:31:00 UTC] {ingest_documents_dag.py:234} INFO - Parsing documents...
[2026-01-25, 10:31:00 UTC] {pdf_parser.py:15} INFO - Parsing PDF: /tmp/ml_orchestration/uploads/1769421678.801241_sample.pdf
[2026-01-25, 10:31:01 UTC] {pdf_parser.py:35} INFO - Extracted 5 pages from PDF
[2026-01-25, 10:31:01 UTC] {ingest_documents_dag.py:250} INFO - Parsed 5 pages from sample.pdf

These logs confirm the DAG is working. If a task fails, the log will show the exception and stack trace.

Now let us verify the data was written to the database.

Verifying Document and Chunk Data in PostgreSQL

The DAG writes to 3 tables: documents, chunks, and pipeline_runs. Let us query them to confirm everything worked.

Connect to PostgreSQL

Open a new terminal and run:

docker exec -it airflow_project-postgres-1 psql -U mlops -d ml_orchestration

This opens a PostgreSQL shell inside the postgres container. You are now connected as the mlops user to the ml_orchestration database.

Query Documents

SELECT id, filename, status, created_at FROM documents;

You should see your document with status = COMPLETED.

Query Chunks

SELECT id, document_id, chunk_index, LENGTH(text) as text_length FROM chunks LIMIT 10;

You should see rows similar to the following:

Each row represents a chunk extracted from the document. The chunk_index indicates the order of chunks within the document. The text_length column shows the character length of each chunk. Since chunking is performed using a target word count (e.g., 512 words with overlap), the character length varies depending on word size and content structure.

Query Pipeline Runs

SELECT run_id, status, documents_processed, chunks_created FROM pipeline_runs;

You should see:

This shows one DAG run processed one document and created 23 chunks. The run_id includes the execution timestamp.

Exit PostgreSQL

Type \q and press Enter to exit the PostgreSQL shell.

You have now verified end-to-end functionality. A document was uploaded, parsed, chunked, validated, and marked complete. The data is in the database and ready for the next stage (embeddings).

Error Handling and Failure Recovery in Airflow Pipeline

Let us walk through a real failure scenario to understand how the system handles errors.

Scenario: A user uploads a corrupted PDF

• Upload Phase: The ingestion service receives the file, computes its hash, and saves it to /tmp/ml_orchestration/uploads/. The file is inserted into the database with status=PENDING. The API returns 201 Created. Everything looks normal.
• Processing Phase: One minute later, the Airflow scheduler triggers the DAG. The fetch_documents task finds the corrupted PDF in the PENDING state and adds its ID to the processing list.
• Parse Failure: The parse_documents task retrieves the file path and calls parse_pdf(). PyPDF attempts to read the file, encounters corrupted data, and raises a PdfReadError. The exception is caught in the try-except block.
• Error Handling: Instead of letting the exception crash the entire task, we log the error, update the document’s status to FAILED, commit the transaction, and continue processing the next document in the batch. Other valid PDFs in the same batch proceed normally.
• Observability: The task completes successfully (from Airflow’s perspective), but logs show which specific document failed and why. The failed document remains in the database with status=FAILED and no chunks. You can query for failed documents and investigate.
• Recovery: Once you fix the corrupted PDF (or replace it), you can manually reset its status back to PENDING. The next DAG run will pick it up and try again.

This granular failure handling is why production systems use orchestrators. One bad file does not bring down the entire pipeline. You get observability into what failed, why it failed, and can fix issues incrementally.

Design Principles: Idempotency, Observability, and Reproducibility in Airflow Pipelines

This architecture demonstrates several important principles for production ML systems.

Separation of Concerns

We separated ingestion from processing. The FastAPI service accepts uploads quickly and returns. It does not block while parsing PDFs or chunking text. Users get fast feedback. Heavy processing happens asynchronously in Airflow.

This separation also means we can scale components independently. If upload traffic spikes, we can run more ingestion service replicas. If processing lags, we can add more Airflow workers.

Idempotency

Every task in the DAG is idempotent. You can rerun the DAG multiple times on the same documents without creating duplicates or corrupting data. This is essential for retries and recovery.

We achieve idempotency through:

• Content hashing: Documents with the same content hash are recognized as duplicates.
• Existence checks: Before creating chunks, we check if they already exist.
• File-based XCom: Parsing results are written to files. If a task reruns, it checks for existing files first.
• Status tracking: Documents have a status field that prevents reprocessing completed items.

Observability

Every operation logs its progress. You can see exactly what the system is doing at any moment. Task logs show which documents were parsed, how many chunks were created, and which operations failed.

The pipeline_runs table provides metrics over time. You can query it to answer questions like “How many documents did we process last week?” or “What is our average chunk count per document?”

Reproducibility

The entire system is defined in code and configuration files. There are no manual setup steps beyond docker compose up. Anyone can clone the repository and run the same system.

The shared module ensures parsing and chunking logic is consistent. Documents processed today will have the same chunk boundaries as documents processed next month.

Failure Handling

The DAG does not abort if one document fails. It marks that document as FAILED and continues with others. This prevents one corrupted PDF from blocking an entire batch.

Airflow’s retry mechanism automatically retries failed tasks. If a transient error (network timeout, database deadlock) causes a failure, the task will retry up to 3 times before giving up.

Data Provenance

We track where every chunk came from. The chunk.document_id and chunk.chunk_index fields let us trace any chunk back to its source document and position. The pipeline_runs table records when and how documents were processed.

This is critical for debugging, auditing, and explaining model behavior. If your RAG system returns an incorrect answer, you can trace it back to the source chunk, the document it came from, and the DAG run that processed it.

Now let’s discuss the limits of this approach.

Limits of Apache Airflow for Machine Learning Workloads

Airflow excels at orchestrating data pipelines, but it has limits for ML workloads.

No GPU Support

Airflow tasks run in Python processes on CPU. If you want to generate embeddings using a transformer model, you need GPU acceleration. Airflow does not provide native GPU scheduling.

You could work around this by calling an external service (like a separate embedding API), but that adds complexity and latency.

Heavy Computation

Embedding generation for thousands of chunks can take minutes or hours. Airflow is not designed for long-running, compute-intensive tasks. Tasks that run for hours can clog the scheduler and make the UI sluggish.

Parallelism

The LocalExecutor runs tasks in parallel on a single machine. If you have 100 documents to process, Airflow will process them sequentially or in small batches limited by your CPU cores.

For true parallelism across multiple machines, you need the CeleryExecutor or KubernetesExecutor, which adds complexity.

Kubernetes Native Workloads

If you are running on Kubernetes, Airflow is an additional system to manage. It needs its own deployment, scheduler, database, and monitoring.

Argo Workflows, which we will cover in the next lesson, is Kubernetes-native. It runs as Kubernetes Jobs and integrates directly with Kubernetes features like resource limits, autoscaling, and GPU scheduling.

When to Use Apache Airflow

Airflow is perfect for:

• Data ingestion and ETL (Extract, Transform, Load)
• Scheduled batch processing
• Orchestrating API calls and database operations
• Tasks that run on CPU and do not require GPU
• Workflows with moderate parallelism

Airflow is not ideal for:

• GPU-accelerated ML training or inference
• Massive parallel processing (thousands of tasks)
• Real-time or low-latency workloads
• Kubernetes-native deployments where you want to avoid managing a separate orchestrator

For our RAG pipeline, Airflow handles document ingestion perfectly. It parses PDFs, chunks text, and validates quality. But for the next stage (generating embeddings), we will switch to Argo Workflows, which can run GPU-accelerated tasks on Kubernetes.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: July 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this lesson, you operationalized the document ingestion architecture built in Lesson 1. You ran the complete system inside Docker, configured environment variables and shared volumes, and verified inter-container communication between FastAPI, Airflow, and PostgreSQL.

You uploaded real PDF documents through the API, triggered DAG executions, monitored task progress in the Airflow UI, inspected logs, and validated results directly in PostgreSQL. You confirmed that documents moved from PENDING to COMPLETED, chunks were created correctly, and pipeline runs were recorded for observability and auditing.

You also explored real-world failure scenarios, including corrupted PDFs and partial processing failures, and saw how the system isolates errors without blocking the entire pipeline. Through this, you reinforced key production principles: separation of concerns, idempotency, observability, reproducibility, and data provenance.

Finally, you examined the practical limits of Apache Airflow for ML workloads and identified where GPU-based execution and Kubernetes-native systems may be more appropriate.

At this point, you now have a fully runnable, production-style ingestion system that you can extend with embedding generation, vector databases, and Retrieval-Augmented Generation workflows.

Citation Information

Singh, V. “Run an Apache Airflow DAG with Docker Compose and PostgreSQL,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/kxc7e

@incollection{Singh_2026_run-apache-airflow-dag-docker-compose-postgresql,
  author = {Vikram Singh},
  title = {{Run an Apache Airflow DAG with Docker Compose and PostgreSQL}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/kxc7e},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Run an Apache Airflow DAG with Docker Compose and PostgreSQL appeared first on PyImageSearch.

Apache Airflow Document Ingestion Pipeline for RAG Systems

In this lesson, you will learn how to design a production-grade document ingestion pipeline using Apache Airflow. We will build a system that accepts PDF uploads via FastAPI and orchestrates their processing using an Airflow DAG (Directed Acyclic Graph). You will see how to structure ingestion pipelines with idempotency, status tracking, and PostgreSQL-backed metadata. By the end of this lesson, you will understand how Airflow fits into modern RAG-style document ingestion workflows.

This lesson is the 1st in a 2-part series on Document Ingestion with Airflow:

1. Apache Airflow Document Ingestion Pipeline for RAG Systems (this tutorial)
2. Lesson 2

To learn how to design and orchestrate a production-ready ingestion pipeline with Apache Airflow, FastAPI, and PostgreSQL, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Introduction to Production-Grade Document Ingestion Pipelines

If you have ever built a Retrieval-Augmented Generation (RAG) system, you know that ingestion is the hardest part. Not the embeddings. Not the vector search. Not even the prompt engineering. The hardest part is reliably getting documents into your system, parsing them correctly, chunking them intelligently, and tracking every step along the way.

Why? Because ingestion is where the real world meets your clean ML architecture. PDFs are corrupted. Files are massive. Network requests fail halfway through. And when something breaks, you need to know exactly which document failed, why it failed, and how to restart processing without duplicating work or losing data.

This is where orchestration becomes critical. You need a system that can schedule work, retry failures, track progress, and give you observability into every stage of your pipeline. For ML ingestion pipelines, Apache Airflow is one of the best tools for this job.

Why Airflow Instead of Cron Jobs or Celery?

You might ask: why not just use cron jobs to trigger a Python script every minute? Or why not use Celery for task queueing? The answer is observability and resilience.

Cron jobs give you scheduling, but no visibility into what failed or why. When a cron job fails at 3am, you find out when users complain. You have no task history, no retry logic, and no dependency management. Celery gives you distributed task execution, but it does not provide workflow orchestration. You have to manually chain tasks, handle retries, and build your own monitoring.

Airflow gives you all of this out of the box. Think of it as a conveyor belt with inspection stations. Every document moves through the same sequence of steps (parse, chunk, validate), and at each station, Airflow records what happened. If a step fails, Airflow retries it automatically. If the entire system crashes, Airflow resumes from where it left off. The web UI shows you exactly which documents are stuck and why.

For production ML systems, this observability is not optional. It is the difference between debugging for hours and knowing immediately which PDF caused the parser to crash.

In this lesson, you will learn how to build a production-grade document ingestion pipeline using Apache Airflow. We will design a complete system that accepts PDF uploads via a REST (Representational State Transfer) API and orchestrates their processing using an Airflow DAG, with full deduplication and idempotency guarantees backed by PostgreSQL.

More importantly, you will understand why Airflow fits ingestion better than training or inference, and where its limitations begin. This foundation prepares you for the next lesson, where we implement the shared parsing and chunking logic and later transition to Argo Workflows for GPU-based ML compute.

By the end of this part, you will have a working control plane for your ingestion pipeline that you can extend for your own RAG systems and document processing workflows.

Let’s get started.

Apache Airflow Document Ingestion Pipeline Architecture

Before we dive into code, let’s understand what we are building. Figure 1 shows the high-level architecture of our ingestion pipeline.

Our system consists of the following 4 main components.

Component 1: FastAPI Ingestion Service

This is the entry point for documents. It exposes a REST API on port 8000 that accepts PDF uploads. When a document arrives, the service performs three critical operations. First, it computes a SHA-256 hash of the file content to detect duplicates. Second, it saves the file to a shared volume that Airflow can access. Third, it inserts a record into the documents table in PostgreSQL with status set to PENDING.

The service does not process the document. It only accepts it and marks it for processing. This separation of concerns is intentional. Ingestion and processing are different responsibilities with different scaling characteristics.

Component 2: Apache Airflow

Airflow is the orchestration layer. It runs two main processes: the scheduler and the webserver. The scheduler monitors our DAG (Directed Acyclic Graph) and triggers it on a schedule. In our case, the DAG runs every minute and looks for documents with status PENDING.

When the DAG runs, it executes a series of tasks in order: fetch pending documents, parse PDFs into pages, chunk the text, validate chunk quality, and mark documents as complete. Each task is idempotent, meaning you can run it multiple times safely. Each task also has retry logic, so transient failures do not require manual intervention.

The webserver provides a UI on port 8080 where you can monitor DAG runs, inspect task logs, and manually trigger runs when needed.

Component 3: PostgreSQL Database

PostgreSQL serves 2 purposes in our system. First, it stores Airflow’s own metadata (DAG runs, task instances, logs). Second, it stores our application data in a separate database called ml_orchestration.

Our application database has 3 main tables. The documents table tracks every uploaded file with its hash, path, and processing status. The chunks table stores the parsed and chunked text with deduplication via content hashes. The pipeline_runs table records every DAG execution with metrics like how many documents were processed and how many chunks were created.

Component 4: Shared Volume

The fourth component is not visible in the diagram, but it is critical. All containers share a Docker volume mounted at /tmp/ml_orchestration/uploads. When the FastAPI service saves a file, Airflow tasks can read it directly without network transfers or complex file synchronization.

This architecture gives us several important properties. First, we have a clear separation between ingestion (FastAPI) and processing (Airflow). Second, we have observability through Airflow’s UI and PostgreSQL queries. Third, we have idempotency through content hashing and status tracking. Fourth, we have reliability through Airflow’s retry mechanisms.

Now let us see how this maps to the actual codebase.

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Project Structure

We first need to review our project directory structure.

Start by accessing this tutorial’s “Downloads” section to retrieve the source code and example images.

From there, take a look at the directory structure:

├── airflow_project/ # Airflow orchestration system
│ ├── dags/
│ │ └── ingest_documents_dag.py # Main DAG: orchestrates PDF→chunks pipeline
│ │
│ ├── ingestion_service/ # FastAPI REST API for file uploads
│ │ ├── __init__.py
│ │ ├── main.py # Upload endpoint with deduplication
│ │ └── requirements.txt # FastAPI, Uvicorn dependencies
│ │
│ ├── docker-compose.yml # Orchestrates 5 services (Postgres, Airflow, API)
│ ├── Dockerfile # Airflow container image
│ ├── Dockerfile.service # FastAPI service container image
│ └── init-db.sh # PostgreSQL database initialization script
│
└── shared/ # Shared utilities (used by Airflow)
├── data_models/
│ ├── __init__.py
│ └── models.py # Pydantic schemas (Document, Chunk, PipelineRun)
│
├── parsing/
│ ├── __init__.py
│ ├── pdf_parser.py # PyPDF extraction logic
│ ├── chunker.py # Sliding window text chunking
│ └── deduplication.py # Content hashing utilities
│
├── storage/
│ ├── __init__.py
│ ├── database.py # SQLAlchemy session management (session_scope, get_session)
│ └── models.py # ORM models (DocumentModel, ChunkModel, PipelineRunModel)
│
├── utils/
│ ├── __init__.py
│ ├── hashing.py # SHA-256 file/content hashing
│ └── logging.py # Structured logging (get_logger)
│
├── __init__.py
└── requirements.txt # Shared dependencies (SQLAlchemy, Pydantic, PyPDF)

Understanding the Structure

This project consists of 2 main directories that work together to create a production-grade document ingestion pipeline.

The airflow_project/ Directory

This folder contains everything for document ingestion using Apache Airflow. Think of it as your document processing factory – where raw PDFs enter the system and emerge as structured, searchable chunks.

The dags/ingest_documents_dag.py file defines our workflow with five sequential tasks: fetch pending documents from the database, parse PDFs with PyPDF, split text into overlapping chunks, validate chunk quality, and mark documents complete. Each task is idempotent (safe to retry) and includes granular error handling so one corrupted PDF does not block an entire batch.

The ingestion_service/ subdirectory runs a FastAPI REST API on port 8000. Users upload PDFs via HTTP POST. The service computes a SHA-256 hash, checks for duplicates, saves the file to a shared volume, and inserts a database record with status=PENDING. It deliberately does not process the file — that separation keeps uploads fast (users get immediate feedback) while heavy processing happens asynchronously in Airflow.

The docker-compose.yml file orchestrates five containers: PostgreSQL (dual purpose: stores Airflow’s metadata and our application data in separate databases), Airflow webserver (UI on port 8080), Airflow scheduler (triggers the DAG every minute), init container (one-time database setup), and the ingestion service (API on port 8000). The critical piece is the shared

/tmp/ml_orchestration/uploads volume mounted into both Airflow containers and the API service – this lets Airflow read files the API writes without network transfers.

The Dockerfile builds the Airflow container with necessary Python dependencies. The Dockerfile.service builds the FastAPI container. The init-db.sh script runs automatically when PostgreSQL starts, creating the ml_orchestration database and mlops user with proper permissions.

The shared/ Directory

This is your reusable logic layer. Everything here is pure Python business logic with zero Airflow dependencies.

The data_models/models.py file contains Pydantic schemas that enforce data structure. Every document has a filename, file path, content hash, and status. Every chunk has text, a content hash, and a document reference. These schemas validate data at the API boundary and prevent type mismatches.

The parsing/ subdirectory implements document processing. The pdf_parser.py module uses PyPDF to extract text page by page, preserving metadata like title and author. The chunker.py module implements sliding window chunking (512 words with 50-word overlap) to split long documents while maintaining context across boundaries. The deduplication.py module computes SHA-256 hashes to detect identical content both at the document and chunk level.

The storage/ subdirectory manages all database interaction. The database.py file provides 2 session management utilities: session_scope() (context manager for Airflow tasks with automatic commit/rollback) and get_session() (generator for FastAPI dependency injection). The models.py file defines SQLAlchemy ORM classes that map Python objects to PostgreSQL tables – DocumentModel, ChunkModel, and PipelineRunModel.

The utils/ subdirectory contains 2 essential helpers. The hashing.py module computes SHA-256 hashes for both files (read in chunks to handle large PDFs) and strings (for chunk deduplication). The logging.py module provides the get_logger() function that returns a configured logger with consistent formatting across the entire system.

Now that you understand where everything lives and why, let’s dive into building the system.

Database Schema Design for Document Ingestion Pipelines

The database schema is the backbone of our ingestion pipeline. Figure 2 shows the 3 main tables and their relationships.

Let’s examine each table and understand the design decisions.

The documents Table

This table tracks every uploaded file. The id column is an auto-incrementing primary key. The filename stores the original name (e.g., research_paper.pdf). The file_path stores the absolute path where the file is saved on disk.

The content_hash column is critical. It stores the SHA-256 hash of the entire file content. This hash serves 2 purposes. First, it detects duplicate uploads. If 2 users upload the same PDF with different filenames, we catch it immediately. Second, it enables idempotency. If we need to reprocess a document, we can verify the file content has not changed.

The status column uses a PostgreSQL ENUM with 4 values: PENDING, PROCESSING, COMPLETED, and FAILED. This drives the entire workflow. The FastAPI service sets status to PENDING. When the DAG completes successfully, Airflow updates it to COMPLETED. If any task fails, it becomes FAILED. (The PROCESSING state is available for systems that want to mark documents as in-progress, though our implementation goes directly from PENDING to COMPLETED or FAILED.)

The created_at and updated_at columns provide audit trails. We know exactly when each document entered the system and when it was last modified.

The chunks Table

This table stores the processed text chunks. The document_id foreign key creates a one-to-many relationship with documents. One document produces many chunks.

The chunk_index tracks the order of chunks within a document. Chunk 0 is the first chunk, chunk 1 is the second, and so on. This ordering is important for maintaining context.

The text column holds the actual chunk content. The content_hash is the SHA-256 of this text. Just like with documents, this prevents duplicate chunks. If the same text appears in multiple places (common in academic papers with repeated abstracts), we store it once.

The page_number tracks which PDF page the chunk came from. This is useful for providing citations back to users. The token_count provides a rough estimate of length (we use word count as a proxy for tokens), which helps with embedding model limits later.

The pipeline_runs Table

This table tracks every DAG execution. The pipeline_type column will eventually distinguish between airflow and argo runs. For now, it is always airflow.

The run_id is Airflow’s unique execution identifier. It looks like manual__2026-01-26T09:56:12.565856+00:00. This connects our table to Airflow’s internal metadata.

The status column tracks whether the entire pipeline run succeeded or failed. The started_at and completed_at timestamps measure execution time.

The metrics columns (documents_processed, chunks_created, embeddings_created) provide observability. You can query this table to see how many documents you have processed over time or track your processing rate.

The run_metadata column is a JSON field for flexible additional data. We store the DAG ID and execution date here.

Why Hashes Matter

Content hashing is not optional in production ML systems. Without hashes, you cannot detect duplicates. Users will upload the same research paper five times, creating 5 sets of chunks and wasting embedding compute and storage.

Without hashes, you cannot implement idempotency. If Airflow retries a task, you might create duplicate chunks or corrupt existing data. With hashes, every operation checks “does this hash already exist?” before creating new records.

Why Idempotency Matters

Idempotency means you can run an operation multiple times and get the same result. This is essential in distributed systems where failures are normal. If your DAG fails halfway through, you should be able to restart it safely.

Our design achieves idempotency through 3 mechanisms. First, content hashes prevent duplicate records. Second, status tracking prevents reprocessing completed documents. Third, task-level checks (e.g., “does this chunk hash already exist?”) ensure partial failures are recoverable.

Database Session Management

Before we dive into the ingestion service and DAG code, we need to understand how we connect to the database. All our code uses 2 key utilities from shared/storage/database.py: session_scope() for Airflow tasks and get_session() for FastAPI.

Here is the complete database connection code:

# shared/storage/database.py
import logging
import os
from contextlib import contextmanager
from typing import Generator
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, Session

logger = logging.getLogger(__name__)

# Base class for all ORM models
Base = declarative_base()

# Database connection string from environment
DATABASE_URL = os.getenv(
    "ML_ORCHESTRATION_DB_URI",
    "postgresql://mlops:mlops_password@localhost:5432/ml_orchestration"
)

def get_engine():
    """
    Create and return a SQLAlchemy engine with connection pooling.
    """
    engine = create_engine(
        DATABASE_URL,
        pool_pre_ping=True,  # Verify connections before using
        pool_size=5,
        max_overflow=10,
        echo=False  # Set to True for SQL query logging
    )
    return engine

@contextmanager
def session_scope():
    """
    Provide a transactional scope for database operations.
    
    Usage in Airflow tasks:
        with session_scope() as session:
            documents = session.query(DocumentModel).all()
    
    This ensures:
    - Automatic commit on success
    - Automatic rollback on exception  
    - Proper connection cleanup
    """
    engine = get_engine()
    SessionLocal = sessionmaker(bind=engine)
    session = SessionLocal()
    try:
        yield session
        session.commit()
    except Exception:
        session.rollback()
        raise
    finally:
        session.close()

def get_session() -> Generator[Session, None, None]:
    """
    FastAPI dependency for database sessions.
    
    Usage:
        @app.post("/documents")
        async def upload(session: Session = Depends(get_session)):
            # Use session here
    
    FastAPI calls this function for each request and handles cleanup.
    """
    engine = get_engine()
    SessionLocal = sessionmaker(bind=engine)
    session = SessionLocal()
    try:
        yield session
    finally:
        session.close()

Let us break down these utilities.

The get_engine() function creates a SQLAlchemy engine, which manages the connection pool to PostgreSQL. The pool_pre_ping=True parameter tells SQLAlchemy to test each connection before using it. This handles cases where the database was restarted or connections went stale.

The pool_size=5 and max_overflow=10 settings control connection pooling. We maintain 5 persistent connections and can create up to 10 additional temporary connections under load. This prevents overwhelming the database with thousands of connections.

The session_scope() context manager is used throughout our Airflow DAG tasks. It provides a transactional scope with automatic cleanup. When you use with session_scope() as session:, the context manager creates a session, executes your code, commits the transaction if successful, or rolls back if an exception occurs. The finally block ensures the connection is always closed.

This pattern prevents common bugs like forgetting to commit, leaking connections, or leaving transactions open after errors.

The get_session() generator is designed for FastAPI’s dependency injection system. FastAPI calls this function for each HTTP request and automatically handles cleanup when the request completes. You never need to manually close the session in your endpoint code.

These 2 utilities ensure database operations are safe, consistent, and clean across both Airflow and FastAPI. Now let us see how the ingestion service uses get_session().

Building a FastAPI Document Ingestion Service

The FastAPI service is the entry point for documents. Let us walk through the code line by line to understand how it works.

Here is the complete upload endpoint from airflow_project/ingestion_service/main.py:

@app.post("/documents", response_model=DocumentResponse, status_code=201)
async def upload_document(
    file: UploadFile = File(...),
    session: Session = Depends(lambda: next(get_session()))
):
    """
    Upload a new document.
    
    The document will be stored and marked as PENDING for processing.
    """
    logger.info(f"Uploading document: {file.filename}")
    
    # Validate file type
    if not file.filename.lower().endswith('.pdf'):
        raise HTTPException(
            status_code=400,
            detail="Only PDF files are supported"
        )
    
    try:
        # Save file to disk
        file_path = UPLOAD_DIR / f"{datetime.utcnow().timestamp()}_{file.filename}"
        
        with open(file_path, "wb") as buffer:
            shutil.copyfileobj(file.file, buffer)
        
        # Compute file hash and size
        content_hash = hash_file(str(file_path))
        file_size = file_path.stat().st_size
        
        # Check for duplicates
        existing_doc = session.query(DocumentModel).filter(
            DocumentModel.content_hash == content_hash
        ).first()
        
        if existing_doc:
            logger.warning(f"Duplicate document detected: {content_hash}")
            file_path.unlink()
            raise HTTPException(
                status_code=409,
                detail=f"Document already exists with ID {existing_doc.id}"
            )
        
        # Create document record
        document = DocumentModel(
            filename=file.filename,
            file_path=str(file_path),
            content_hash=content_hash,
            file_size=file_size,
            mime_type="application/pdf",
            status=DocumentStatus.PENDING
        )
        
        session.add(document)
        session.commit()
        session.refresh(document)
        
        logger.info(f"Document uploaded successfully: ID {document.id}")
        
        return DocumentResponse.from_orm(document)
        
    except HTTPException:
        raise
    except Exception as e:
        logger.error(f"Failed to upload document: {str(e)}")
        raise HTTPException(status_code=500, detail=str(e))

Let us break this down step by step.

The function signature uses FastAPI’s dependency injection. The file parameter comes from the HTTP request as multipart form data. The session parameter is injected by FastAPI using Depends(). This gives us a database session without manual connection management.

The first operation is file type validation. We only accept PDFs for this lesson, so we check the filename extension. If it is not a PDF, we raise an HTTP 400 error immediately. Production systems might also validate file size, scan for malware, or check MIME types, but we keep it simple here.

Next, we save the file to disk. The UPLOAD_DIR is /tmp/ml_orchestration/uploads. This directory is mounted as a Docker volume, which means all containers can access it. We prefix the filename with a UTC timestamp to avoid collisions. If 2 users upload files named paper.pdf, they become 1769421678.801241_paper.pdf and 1769421690.123456_paper.pdf.

We use shutil.copyfileobj() to stream the file content from the upload to disk. This is memory-efficient because it processes the file in chunks rather than loading the entire file into RAM.

After saving, we compute 2 important values. The hash_file() function reads the entire file and computes its SHA-256 hash. This is a cryptographic hash function that produces a unique 64-character hexadecimal string for the file content. Even a single byte change produces a completely different hash. We also get the file size in bytes using file_path.stat().st_size.

The next step is critical: duplicate detection. We query the database for any existing document with the same content hash. If we find one, we know this exact file has been uploaded before, even if it has a different filename. We delete the newly uploaded file with file_path.unlink() and return an HTTP 409 Conflict error with the ID of the existing document. This prevents duplicate processing.

If the document is unique, we create a new DocumentModel instance. Notice the status field is set to DocumentStatus.PENDING. This tells Airflow that the document needs processing. We do not set it to PROCESSING or COMPLETED because the upload service does not process documents. It only accepts them.

We add the model to the session, commit the transaction, and refresh the model to get the auto-generated ID. Finally, we return a DocumentResponse with all the document details. The HTTP status code is 201 Created, which is the correct status for successful resource creation.

The error handling is worth noting. We re-raise HTTPException instances without modification because FastAPI knows how to convert them to HTTP responses. For all other exceptions, we log the error and return an HTTP 500 with the error message. In production, you would want more sophisticated error handling (do not expose internal errors to clients), but this is sufficient for a lesson.

What This Service Does Not Do

Notice what is missing from this code. There is no PDF parsing. No text chunking. No embedding generation. The service has one responsibility: accept files and mark them for processing. This separation is intentional.

Ingestion and processing are different concerns. Ingestion must be fast and available. Users should be able to upload files without waiting for heavyweight processing. Processing can happen asynchronously, can retry on failure, and can take as long as needed.

This is where Airflow enters the picture. Let us see how the DAG processes these pending documents.

Designing an Apache Airflow DAG

The DAG is the heart of our orchestration logic. Figure 4 shows the task graph and execution order.

Here is how the DAG is defined in airflow_project/dags/ingest_documents_dag.py:

with DAG(
    dag_id='ingest_documents_dag',
    default_args=default_args,
    description='Ingest and process documents for ML pipeline',
    schedule_interval=timedelta(minutes=1),
    start_date=days_ago(1),
    catchup=False,
    tags=['ingestion', 'documents', 'ml-pipeline'],
) as dag:
    
    fetch_documents_task = PythonOperator(
        task_id='fetch_documents',
        python_callable=fetch_pending_documents,
        provide_context=True,
    )
    
    parse_documents_task = PythonOperator(
        task_id='parse_documents',
        python_callable=parse_documents,
        provide_context=True,
    )
    
    chunk_documents_task = PythonOperator(
        task_id='chunk_documents',
        python_callable=chunk_documents,
        provide_context=True,
    )
    
    validate_chunks_task = PythonOperator(
        task_id='validate_chunks',
        python_callable=validate_chunks,
        provide_context=True,
    )
    
    mark_complete_task = PythonOperator(
        task_id='mark_complete',
        python_callable=mark_documents_complete,
        provide_context=True,
    )
    
    # Define task dependencies
    fetch_documents_task >> parse_documents_task >> chunk_documents_task
    chunk_documents_task >> validate_chunks_task >> mark_complete_task

Let us understand each configuration parameter.

The dag_id is the unique identifier for this workflow. It appears in the Airflow UI and logs. The default_args dictionary contains settings that apply to all tasks. This includes retry behavior, execution timeout, and owner information.

The schedule_interval is set to timedelta(minutes=1). This means Airflow runs this DAG every minute. In production, you might use hourly or daily schedules, but for demos and development, 1 minute lets you see results quickly.

The start_date is set to days_ago(1), which means the DAG is eligible to run starting yesterday. The catchup=False parameter is important. Without this, Airflow would try to run the DAG for every missed interval since the start date. We do not want that. We only care about processing current pending documents, not creating historical backfill runs.

The tags list helps organize DAGs in the UI. You can filter by tag to find related workflows.

Each task uses a PythonOperator, which executes a Python function. The task_id must be unique within the DAG. The python_callable is the function to execute. The provide_context=True parameter gives the function access to Airflow’s execution context.

Why does context matter? Because it provides critical runtime information: the unique run_id (for creating file names that do not collide across runs), the execution timestamp (for audit trails), and XCom access (for passing data between tasks). Without context, your task functions would be isolated and unable to coordinate or share state.

The task dependencies are defined using the >> operator. This creates a directed graph. fetch_documents_task >> parse_documents_task means parse documents cannot start until fetch documents completes. The final line creates a longer chain: fetch, then parse, then chunk, then validate, then mark complete. This ensures strict ordering.

Now let us examine what each task function does.

Task 1: Fetch Pending Documents

def fetch_pending_documents(**context) -> List[int]:
    """
    Task 1: Fetch documents that need processing.
    
    Returns list of document IDs to process.
    """
    logger.info("Fetching pending documents...")
    
    with session_scope() as session:
        pending_docs = session.query(DocumentModel).filter(
            DocumentModel.status == DocumentStatus.PENDING
        ).all()
        
        doc_ids = [doc.id for doc in pending_docs]
        logger.info(f"Found {len(doc_ids)} pending documents: {doc_ids}")
        
        run_id = context['dag_run'].run_id
        filepath = write_data_to_file(doc_ids, f'{run_id}_document_ids.json')
        
        context['task_instance'].xcom_push(key='document_ids_file', value=filepath)
        
        return doc_ids

This function queries the database for all documents where status = PENDING. It extracts just the IDs into a list. If there are no pending documents, the list is empty and subsequent tasks have no work to do.

The interesting part is how we pass data to the next task. We do not use Airflow’s XCom directly for the document IDs. Instead, we write them to a JSON file and pass only the file path through XCom. Why? Because XCom stores data in the Airflow metadata database. Large payloads slow down the database and can hit size limits. By using files, we keep XCom small and handle arbitrary data sizes.

The write_data_to_file() helper function writes JSON to /tmp/***_dag_data/ and returns the full path. The next task reads from this path.

Task 2: Parse Documents

def parse_documents(**context) -> Dict[str, int]:
    """
    Task 2: Parse PDF documents into pages.
    
    Reads document IDs from previous task and parses each PDF.
    """
    logger.info("Parsing documents...")
    
    doc_ids_file = context['task_instance'].xcom_pull(
        key='document_ids_file',
        task_ids='fetch_documents'
    )
    doc_ids = read_data_from_file(doc_ids_file)
    
    parsed_count = 0
    
    with session_scope() as session:
        for doc_id in doc_ids:
            pages_file = TEMP_DIR / f'{run_id}_doc_{doc_id}_pages.json'
            if pages_file.exists():
                logger.info(f"Document {doc_id} already parsed, skipping")
                parsed_count += 1
                continue
            
            doc = session.query(DocumentModel).filter(
                DocumentModel.id == doc_id
            ).first()
            
            if not doc:
                logger.warning(f"Document {doc_id} not found")
                continue
            
            try:
                pages = parse_pdf(doc.file_path)
                logger.info(f"Parsed {len(pages)} pages from {doc.filename}")
                
                pages_file = write_data_to_file(pages, f'{run_id}_doc_{doc_id}_pages.json')
                parsed_count += 1
                
            except Exception as e:
                logger.error(f"Failed to parse document {doc_id}: {str(e)}")
                doc.status = DocumentStatus.FAILED
                session.commit()
    
    logger.info(f"Successfully parsed {parsed_count} documents")
    return {'parsed': parsed_count}

This task pulls the document IDs from the previous task, loads the document record from the database, and calls parse_pdf() on the file path. The parse_pdf() function (from shared/parsing/pdf_parser.py) uses PyPDF to extract text page by page.

Notice the idempotency check at the top of the loop. If a file named {run_id}_doc_{doc_id}_pages.json already exists, we skip parsing. This means if the task retries or reruns, it does not waste time reparsing documents that succeeded before.

The error handling is important. If parsing fails for any reason (corrupted PDF, missing file, permission error), we catch the exception, mark that document as FAILED, and continue with the next one. This prevents one bad document from blocking the entire batch.

The parsed pages are written to a file, one file per document. The next task will read these files.

Task 3: Chunk Documents

def chunk_documents(**context) -> Dict[str, int]:
    """
    Task 3: Chunk parsed pages into text segments.
    
    Reads pages from previous task and creates chunks.
    """
    logger.info("Chunking documents...")
    
    doc_ids_file = context['task_instance'].xcom_pull(
        key='document_ids_file',
        task_ids='fetch_documents'
    )
    doc_ids = read_data_from_file(doc_ids_file)
    
    total_chunks = 0
    
    with session_scope() as session:
        for doc_id in doc_ids:
            run_id = context['dag_run'].run_id
            pages_file = TEMP_DIR / f'{run_id}_doc_{doc_id}_pages.json'
            
            if not pages_file.exists():
                logger.warning(f"No pages file found for document {doc_id}")
                continue
            
            pages = read_data_from_file(str(pages_file))
            
            existing_chunks = session.query(ChunkModel).filter(
                ChunkModel.document_id == doc_id
            ).count()
            
            if existing_chunks > 0:
                logger.info(f"Document {doc_id} already has {existing_chunks} chunks, skipping")
                total_chunks += existing_chunks
                continue
            
            try:
                full_text = "\n\n".join(page['text'] for page in pages)
                chunks = chunk_text(full_text, chunk_size=512, overlap=50)
                
                chunk_index = 0
                for chunk in chunks:
                    chunk_hash = hash_content(chunk)
                    
                    existing_chunk = session.query(ChunkModel).filter(
                        ChunkModel.content_hash == chunk_hash
                    ).first()
                    
                    if existing_chunk:
                        continue
                    
                    chunk_model = ChunkModel(
                        document_id=doc_id,
                        chunk_index=chunk_index,
                        text=chunk,
                        content_hash=chunk_hash,
                        page_number=None,
                        token_count=len(chunk.split())
                    )
                    session.add(chunk_model)
                    chunk_index += 1
                
                session.commit()
                logger.info(f"Created {chunk_index} chunks for document {doc_id}")
                total_chunks += chunk_index
                
            except Exception as e:
                logger.error(f"Failed to chunk document {doc_id}: {str(e)}")
    
    logger.info(f"Total chunks created: {total_chunks}")
    return {'chunks': total_chunks}

This task joins all pages into a single text string, then calls chunk_text() to split it into overlapping segments. The default chunk size is 512 words (we use whitespace-separated words as an approximate proxy for tokens) with 50-word overlap. Think of this like cutting a long rope into segments with intentional overlap at the ends — if an important concept spans a boundary, the overlap ensures it appears fully in at least one segment.

For each chunk, we compute a content hash and check if that exact text already exists in the database. This is duplicate detection at the chunk level. If the same sentence appears in multiple documents, we store it once. This saves storage and embedding compute later.

Notice we track chunk_index to maintain ordering within a document. This is important for reconstruction or citation purposes.

The task again has idempotency checks. If the document already has chunks in the database, we skip it. This lets us safely retry the task.

Task 4: Validate Chunks

def validate_chunks(**context) -> Dict[str, int]:
    """
    Task 4: Validate chunk quality.
    
    Checks for empty chunks, excessive length, etc.
    """
    logger.info("Validating chunks...")
    
    doc_ids_file = context['task_instance'].xcom_pull(
        key='document_ids_file',
        task_ids='fetch_documents'
    )
    doc_ids = read_data_from_file(doc_ids_file)
    
    valid_count = 0
    invalid_count = 0
    
    with session_scope() as session:
        for doc_id in doc_ids:
            chunks = session.query(ChunkModel).filter(
                ChunkModel.document_id == doc_id
            ).all()
            
            for chunk in chunks:
                # Too short
                if len(chunk.text) < 50:
                    logger.warning(f"Chunk {chunk.id} too short: {len(chunk.text)} chars")
                    invalid_count += 1
                    continue
                
                # Too long
                if len(chunk.text) > 2000:
                    logger.warning(f"Chunk {chunk.id} too long: {len(chunk.text)} chars")
                    invalid_count += 1
                    continue
                
                # Empty or whitespace only
                if not chunk.text.strip():
                    logger.warning(f"Chunk {chunk.id} is empty or whitespace only")
                    invalid_count += 1
                    continue
                
                valid_count += 1
    
    logger.info(f"Validation complete: {valid_count} valid, {invalid_count} invalid chunks")
    return {'valid': valid_count, 'invalid': invalid_count}

This task performs quality checks on chunks. It checks for chunks that are too short (less than 50 characters), too long (more than 2000 characters), or empty. In production, you might delete invalid chunks or mark them in a separate table. Here, we just log warnings.

Task 5: Mark Complete

def mark_documents_complete(**context) -> Dict[str, int]:
    """
    Task 5: Mark documents as complete.
    
    Updates document status and creates pipeline run record.
    """
    logger.info("Marking documents complete...")
    
    doc_ids_file = context['task_instance'].xcom_pull(
        key='document_ids_file',
        task_ids='fetch_documents'
    )
    doc_ids = read_data_from_file(doc_ids_file)
    
    chunks_result = context['task_instance'].xcom_pull(task_ids='chunk_documents')
    total_chunks = chunks_result.get('chunks', 0)
    
    with session_scope() as session:
        for doc_id in doc_ids:
            doc = session.query(DocumentModel).filter(
                DocumentModel.id == doc_id
            ).first()
            
            if doc and doc.status == DocumentStatus.PROCESSING:
                doc.status = DocumentStatus.COMPLETED
        
        run_id = context['dag_run'].run_id
        pipeline_run = PipelineRunModel(
            pipeline_type='airflow',
            run_id=run_id,
            status=PipelineRunStatus.COMPLETED,
            started_at=context['dag_run'].start_date,
            completed_at=datetime.utcnow(),
            documents_processed=len(doc_ids),
            chunks_created=total_chunks,
            embeddings_created=0,
            run_metadata={
                'dag_id': context['dag'].dag_id,
                'execution_date': str(context['execution_date'])
            }
        )
        
        session.add(pipeline_run)
        session.commit()
        
        logger.info(f"Pipeline run {run_id} completed: {len(doc_ids)} docs, {total_chunks} chunks")
    
    return {'documents_completed': len(doc_ids)}

The final task updates document status to COMPLETED and creates a PipelineRunModel record. This record captures metrics about the entire DAG run. Later, you can query this table to track throughput, find bottlenecks, or generate reports.

Why This DAG Structure Works

This 5-task structure enforces 4 critical principles.

First, each task has a single responsibility. Fetch finds work. Parse extracts text. Chunk splits text. Validate checks quality. Mark complete updates status. This makes debugging easier. If chunking fails, you know exactly which task to inspect.

Second, each task is idempotent. You can retry tasks without creating duplicate data or corrupting state. This is essential for reliability.

Third, we have observability at every step. Each task logs its progress. You can see exactly how many documents were parsed, how many chunks were created, and which documents failed.

Fourth, failure handling is granular. The pipeline is designed to continue processing other documents when individual documents fail, rather than aborting the entire batch. We catch exceptions at the document level, mark failed documents with FAILED status, and let the task continue with the remaining documents.

In an upcoming lesson, we will implement the shared parsing and chunking logic and see how these tasks operate on real documents end to end.

What's next? We recommend PyImageSearch University.

Course information:
86+ total classes • 115+ hours hours of on-demand code walkthrough videos • Last updated: July 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 86+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 86 Certificates of Completion
• ✓ 115+ hours hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this lesson, you built the foundation of a production-grade document ingestion pipeline using Apache Airflow. You learned how to design a FastAPI service for uploading PDF documents with built-in deduplication, how to model ingestion state in PostgreSQL, and how to define a reliable Airflow DAG to orchestrate document processing.

You saw how to separate ingestion from processing, use content hashing for idempotency, and construct a task graph that represents each stage of the pipeline. By the end of this part, you had a complete orchestration design for moving documents from raw uploads into a scheduled workflow.

This architecture forms the control plane of your ingestion pipeline and prepares you to implement the parsing and chunking logic in the next part.

Citation Information

Singh, V. “Apache Airflow Document Ingestion Pipeline for RAG Systems,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/8b2ey

@incollection{Singh_2026_apache-airflow-document-ingestion-pipeline-rag-systems,
  author = {Vikram Singh},
  title = {{Apache Airflow Document Ingestion Pipeline for RAG Systems}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/8b2ey},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Apache Airflow Document Ingestion Pipeline for RAG Systems appeared first on PyImageSearch.