YOLO26 Open-Vocabulary Object Detection with YOLOE-26

In this lesson, you will learn how YOLOE evolved into YOLOE-26, how open-vocabulary detection works, and how to use text prompts, visual prompts, and prompt-free inference with Ultralytics.

This lesson is the 1st in a 2-part series on YOLOE-26 and open-vocabulary detection:

1. YOLO26 Open-Vocabulary Object Detection with YOLOE-26 (this tutorial)
2. Lesson 2

To learn how to use YOLOE-26 for open-vocabulary object detection with text prompts, visual prompts, and prompt-free inference, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

In the last YOLO26 lesson, we stayed in the closed-set world. We loaded a modern YOLO detector, ran it on images and video, and saw just how fast and polished the Ultralytics pipeline has become.

But closed-set detection has a hard limit. A model can only detect the classes it learned during training.

That sounds obvious until you hit it in practice.

You may want to detect a multimeter, a barcode scanner, a microscope slide box, a soldering iron, or a specific logo in a warehouse photo. A standard YOLO model cannot suddenly understand those categories just because you typed their names. If the class was not part of training, the model either misses it or maps it to the closest thing it already knows.

That is the problem YOLOE was built to solve.

This lesson is the foundation piece for the two-part series. We are not fine-tuning anything yet. We are first building the mental model you need before the workflow gets more ambitious in Lesson 2.

In this lesson, you will learn:

• what open-vocabulary detection actually means in plain language
• what YOLOE introduced to the YOLO family
• how YOLOE-26 extends that idea into the YOLO26 generation
• how to run text-prompted, visual-prompted, and prompt-free inference
• when YOLOE-26 is the right tool and when a standard YOLO26 model is still the better choice

Understanding Closed-Set YOLO Object Detection

Let us begin with the constraint that defines standard YOLO models.

A regular YOLO26 detector is excellent at recognizing the categories it was trained on. But it still operates inside a fixed label space. If the class is outside that label space, the model has no mechanism to dynamically add it at inference time.

That is the itch this lesson scratches.

The first example below runs a standard YOLO26 detector on the familiar bus.jpg sample image that ships with Ultralytics:

from ultralytics import YOLO
from ultralytics.utils import ASSETS

baseline_model = YOLO("yolo26s.pt")
baseline_results = baseline_model.predict(ASSETS / "bus.jpg", conf=0.25)
baseline_results[0].show()

In that image, YOLO26 does exactly what you would expect. It finds the obvious closed-set classes such as people and the bus itself.

The problem shows up when your application stops looking like a benchmark and starts looking like the real world. In production, the target object list is often messy, narrow, and unstable. Maybe your robotics pipeline needs to find a clamp that appears in only one manufacturing line. Maybe your e-commerce workflow needs to find a specific product family whose packaging changed last quarter. Maybe your lab automation setup needs to distinguish one kind of equipment tray from another.

In all of those scenarios, a closed-set model puts you in one of 2 boxes:

• the class already exists in the model, so you are fine
• the class does not exist, so you need a new data collection and training workflow

That second path is expensive. You need images. You need labels. You need training time. You need evaluation. You probably need iteration because the first pass will not be clean enough.

This is why open-vocabulary detection matters. It does not replace training forever, but it gives you a much more flexible first step.

This is the key mental shift for the rest of the article. YOLOE does not just make YOLO “a bit better.” It changes how you specify what the model should look for.

Where YOLO Fits Among Object Detection Models

Before going further, it helps to know where YOLO sits among object detection models generally, since not every detector works the same way under the hood.

Most object detection models fall into 1 of 2 architectural families. Two-stage detectors (e.g., Faster Region-based Convolutional Neural Network (R-CNN) implementations available through Detectron2) first propose candidate regions in an image, then classify each region separately. This tends to produce strong accuracy, but the two-pass design costs speed. One-stage detectors, the family YOLO belongs to, predict boxes and classes in a single forward pass. That speed advantage helped make YOLO a common choice for real-time applications, historically trading some accuracy for substantially faster inference.

That architectural split is only half the picture, though, and it is the half most tutorials stop at. A second, independent axis matters just as much for this lesson: whether the class list is closed or open. Whether a detector is one-stage or two-stage, the overwhelming majority share the same constraint: a fixed set of classes, usually drawn from a benchmark like Common Objects in Context (COCO), locked in at training time. To detect anything outside that list, you have to collect new images, label them, and train an object detection model from scratch on the updated data. That process works, but it is slow, and it has to be repeated every time your target categories change.

YOLO26 is a fast, one-stage, closed-set detector. It is excellent at what it was trained to recognize but is bound by that same COCO-style constraint everywhere else. YOLOE-26 keeps YOLO’s one-stage speed but breaks the second constraint. That is the shift the rest of this lesson is about.

Why Open-Vocabulary and Zero-Shot Object Detection Matter

Before we touch YOLOE itself, let us make the term open vocabulary concrete.

In a closed-set detector, the label list is fixed ahead of time. In an open-vocabulary detector, the model can work from language prompts, visual references, or a built-in broader vocabulary instead of a single hardcoded class list.

The practical benefit is simple: you can ask the detector for new concepts without rebuilding the whole model every time the problem changes. That is especially useful when the object is outside everyday benchmark classes, when the label list changes often, or when you want to bootstrap a later fine-tuning workflow.

This is closely related to what is called zero-shot detection in the research literature: the ability to detect object categories the model was never explicitly trained on. Open-vocabulary detection is the broader, more flexible version of that idea, built around accepting arbitrary prompts at inference time rather than a fixed set of unseen classes.

Earlier systems (e.g., Grounded Language-Image Pre-training (GLIP), Open-World Localization Vision Transformer (OWL-ViT), and Grounding DINO) proved that open-vocabulary detection works, but they also made clear how expensive the usual vision-language path can become.

YOLOE matters because it tries to keep that promptable behavior while staying in the real-time YOLO regime.

For PyImageSearch readers who have already trained detectors before, the right framing is this: YOLOE-26 is not a replacement for every older YOLO workflow. It is a new capability layer on top of familiar YOLO deployment patterns.

What YOLOE Introduced

YOLOE, introduced in the paper YOLOE: Real-Time Seeing Anything, takes the familiar YOLO workflow and adds open-vocabulary behavior on top of it.

Instead of being locked to a fixed class list, YOLOE can operate through 3 different prompting modes:

• Text prompting: tells the model what classes to look for using words
• Visual prompting: shows the model an example object and asks it to find more like it
• Prompt-free mode: uses a built-in open vocabulary without supplying your own prompts at inference time

This is the core idea you should carry forward. YOLOE is still part of the YOLO family. It still feels like Ultralytics. But it replaces the “fixed class list only” assumption with a more flexible prompting interface.

The reason that matters is simple. In a text-prompted setting, the detector is no longer just asking, “Does this image contain one of the classes already in its label space?” It is also asking, “How well does this region align with the prompt concept it was given?” That is the conceptual jump.

You do not need to understand every detail of the paper to benefit from it. For this lesson, the important point is that YOLOE creates a bridge between image features and promptable concepts. Sometimes those concepts come from text. Sometimes they come from visual examples. Sometimes they come from a prebuilt broader vocabulary.

At this point, the most useful question is not “How does every module work?” It is “What new things can we now ask the model to do?” The answer is that YOLOE gives you multiple ways to specify the object of interest, which is exactly what a closed-set detector lacks.

How YOLOE-26 Extends Open-Vocabulary Detection to YOLO26

Earlier YOLOE releases were shipped in YOLOv8-based and YOLO11-based model families. YOLOE-26 brings the same open-vocabulary idea into the YOLO26 family.

The following 2 points matter here. YOLO26 introduced a cleaner end-to-end design with native non-maximum suppression (NMS)-free inference, simpler heads, and a stronger accuracy-latency balance. YOLOE-26 inherits that foundation while keeping the promptable modes that made YOLOE interesting in the first place.

Architecturally, it still follows the familiar YOLO pattern: a backbone for feature extraction, a neck for multi-scale fusion, and a prediction head that turns fused features into boxes, masks, and region-level representations.

According to the Ultralytics docs, YOLOE-26:

• extends the YOLOE family onto the YOLO26 backbone
• inherits the NMS-free end-to-end design of YOLO26
• supports text prompting, visual prompting, and prompt-free inference
• is available across 5 scales: n, s, m, l, and x

Model Family and Checkpoints

The naming convention is easy to miss when you first see the model files:

• yoloe-26n-*: nano variant
• yoloe-26s-*: small variant
• yoloe-26m-*: medium variant
• yoloe-26l-*: large variant
• yoloe-26x-*: extra-large variant

For this lesson, we are using the s scale so the workflow stays light enough to reproduce easily. We are also using -seg checkpoints because the overlays are easier to inspect visually and align well with the official Ultralytics examples. The released pretrained YOLOE checkpoints are segmentation-first, which is why the lesson uses segmentation weights even when the conceptual discussion focuses on detection behavior.

Choosing the Right Checkpoint

In practice, there are really 2 checkpoint families to keep straight:

• yoloe-26*-seg.pt: for text prompting and visual prompting
• yoloe-26*-seg-pf.pt: for prompt-free inference

The scale suffix then controls the usual size-speed-accuracy tradeoff:

• n: for the smallest and lightest deployment target
• s: for a practical small-model starting point
• m and l: when you can spend more compute for stronger quality
• x: when you want the highest-end model in the family

For Lesson 1, the s variant is the right teaching checkpoint. It is large enough to show the behavior clearly, but still light enough that readers can reproduce the examples without needing a heavyweight setup. The segmentation-first checkpoints also help because masks make qualitative inspection easier, even when our discussion is mostly about the detection logic.

This is also where Lesson 2 starts to come into focus. We are not reproducing the full YOLOE training pipeline here, but we will borrow the same high-level idea later: use an open-vocabulary model to help create labels for a narrower downstream detector.

The following 3 architecture details are worth calling out while that figure is on screen:

• Re-parameterizable Region-Text Alignment (RepRTA): supports text-prompted detection.
• Semantic-Activated Visual Prompt Encoder (SAVPE): supports visual-prompted detection.
• Lazy Region-Prompt Contrast (LRPC): supports prompt-free open-vocabulary inference.

You do not need to memorize those names. The practical takeaway is that YOLOE adds dedicated machinery for each prompting mode while preserving the familiar YOLO deployment feel.

How the YOLOE-26 Flow Works

At this point, it helps to slow down and build a slightly more mechanical mental model of what is happening inside the model.

At a high level, YOLOE-26 still begins like a normal YOLO pipeline. The input image goes through a backbone that extracts visual features, then through a neck that fuses those features across scales so small, medium, and large objects can all be represented well. Up to that point, the story still feels very familiar to anyone who has used YOLO before.

The open-vocabulary twist happens after those visual features have been built.

Conceptually, you can think of YOLOE-26 as doing 5 stages:

• Input image to visual features: the image is converted into multi-scale feature maps by the backbone.
• Feature fusion: the neck combines information across scales so object evidence is available at multiple resolutions.
• Region prediction: the model predicts boxes and, in the segmentation checkpoints used here, masks as well.
• Prompt alignment: the model also needs a way to compare each candidate region against some notion of “what you are asking for.”
• Similarity scoring to final detections: region-level visual features are matched against prompt embeddings, and the best matches become named detections.

Another useful way to think about the head is this: boxes and masks still come from a normal YOLO-style prediction path, but class scoring is no longer a fixed set of logits over a closed label list. Instead, each candidate region is compared against prompt embeddings, and that similarity score takes over the role that fixed class scores play in a standard detector.

If you want a simple intuition, think of standard YOLO as answering, “Where are the objects already in the label space?” YOLOE-26 answers two questions at once: “Where are the candidate objects?” and “Which of these candidate regions best matches the prompt concept it was given?”

One important note: the simplified flow diagram below is a conceptual teaching figure, not a literal one-to-one tensor graph from the paper. That is intentional. For a blog lesson, the goal is to make the pipeline intuitive before readers dive into implementation details.

What Each Prompt Mode Changes

The cleanest way to understand the architecture is to ask what changes between the 3 prompting modes.

For text prompting, YOLOE uses RepRTA. According to the YOLOE paper, this module refines pretrained text embeddings through a lightweight auxiliary network so the prompt representation aligns better with the detector’s visual region features. The important deployment detail is that this lightweight refinement can be folded back into the model at inference time, which is why text prompting does not introduce the kind of heavy runtime penalty many vision-language models do.

For visual prompting, YOLOE uses SAVPE. Instead of starting from words, it starts from a reference object. The prompt encoder uses semantic and activation cues from that reference so the detector can look for visually similar regions elsewhere. That is why visual prompting feels closer to one-shot retrieval than to ordinary closed-set classification.

For prompt-free mode, YOLOE uses LRPC. Here the model does not wait for an external text prompt at all. It uses a built-in vocabulary and specialized internal embeddings, then scores candidate regions against that internal vocabulary. The Ultralytics docs describe this as open-set recognition using internal embeddings trained on large vocabularies, which is what lets prompt-free mode run without an external prompt encoder at inference time.

This also explains an important checkpoint detail for readers. Text and visual prompting use the same main YOLOE checkpoint family, while prompt-free mode uses separate -pf weights because those models are trained as built-in large-vocabulary variants rather than prompt-conditioned ones.

Where YOLO26 Changes the Base

Now add the YOLO26 side of the story.

The earlier YOLOE families were built on prior YOLO backbones, and YOLOE-26 inherits the lighter, native end-to-end design of YOLO26. According to the YOLO26 paper, that includes NMS-free end-to-end inference, a lighter head with Distribution Focal Loss (DFL) removed, and a training recipe designed to better match the inference-time head. In practice, that means YOLOE-26 is not just “YOLOE with a new name.” It is YOLOE running on a cleaner deployment-oriented detector backbone.

The key intuition is this: YOLOE contributes the promptable alignment machinery, and YOLO26 contributes the faster, simpler end-to-end detector base.

How YOLOE-26 Trains for Open-Vocabulary Object Detection

Readers usually do not need the full training recipe, but they do benefit from understanding what is trained differently.

How Text, Visual, and Prompt-Free Training Diverge

For text prompting, the paper explains that pretrained text embeddings are refined by a lightweight auxiliary network before being aligned with visual region features. That refinement step is a big part of why YOLOE can work with prompts effectively without dragging a large language branch through the full deployment path.

For visual prompting, the model learns how to turn a reference object into a useful prompt representation rather than treating the crop as a raw patch match. That is what SAVPE is doing conceptually: learning a compact visual prompt that can be compared against candidate regions.

For prompt-free mode, the model is trained to work against a built-in vocabulary and internal embedding space, so it can still perform open-vocabulary recognition even when no external prompt is supplied.

Again, the lesson-level takeaway is not every training detail. The takeaway is that YOLOE is trained to bring regions and prompts into a comparable embedding space, then YOLOE-26 places that behavior on top of the more deployment-friendly YOLO26 detector.

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Configuring Your Development Environment

To follow this guide, you need to install the Ultralytics package.

Luckily, Ultralytics is pip-installable:

$ pip install -U ultralytics

If you want to be a little safer for local runs, you can use:

$ pip install -U ultralytics opencv-python matplotlib

If you need help configuring your development environment for OpenCV, we highly recommend reading our pip install OpenCV guide. It will have you up and running in minutes.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

YOLOE-26 Object Detection Benchmarks and Performance

Before writing code, let us calibrate expectations.

Ultralytics reports that YOLOE-L and YOLOE26-L preserve near-identical inference speed to their underlying closed-set counterparts, while adding open-vocabulary capability. The same documentation also reports stronger Large Vocabulary Instance Segmentation (LVIS) open-vocabulary performance for the YOLO26-based branch.

The important pattern is not just that YOLOE-26 scores higher on LVIS. It does so while preserving the same reported T4 latency as YOLO11-L and YOLOE-L, which is exactly why the open-vocabulary story is practical rather than just academic.

The docs also make an important practical claim: in the regular closed-set case, the open-world additions in YOLOE can be re-parameterized back into a standard YOLO-style path, so you do not pay extra inference cost just for carrying the capability.

It is also useful to unpack the benchmark language briefly so readers do not treat the numbers like magic:

• LVIS is a long-tail detection benchmark, which makes it a more meaningful place to discuss open-vocabulary behavior than a short everyday-class benchmark
• AP is average precision, so higher is better
• T4 latency gives you a rough sense of runtime cost on a standard GPU reference point

The benchmark story here is narrower and more useful than hype. YOLOE-26 improves the open-vocabulary side of the problem while staying in the performance neighborhood practitioners expect from modern YOLO models.

Hands-On with Text Prompting

Now let us move from concept to code.

The most beginner-friendly way to meet YOLOE-26 is through text prompting. You load a YOLOE model, call set_classes() once, and then run predict() just like you would with a normal Ultralytics model.

from ultralytics import YOLOE
from ultralytics.utils import ASSETS

text_model = YOLOE("yoloe-26s-seg.pt")
text_model.set_classes(["person", "bus"])
text_results = text_model.predict(ASSETS / "bus.jpg", conf=0.25)
text_results[0].show()

The following 2 points are worth noticing. The application programming interface (API) is almost boringly simple, and the class list is no longer hardwired into the checkpoint in the same way a standard closed-set detector is. You are telling the model what to care about at inference time.

In this first text-prompt example, we are using person and bus because they are stable and easy to reproduce in the official sample image. Once you understand the workflow, you can swap in more interesting prompts that match your own data.

That basic example is intentionally conservative. It is not trying to prove that YOLOE-26 magically does something a closed-set COCO model never could. It is showing you the mechanics in the simplest reproducible way.

Once that is clear, the real value comes from changing the prompt list.

How to Think About Prompt Design

This is where readers often make a subtle mistake. They treat prompting like keyword search. Open-vocabulary detection is not just string matching, so prompt quality matters.

A few practical rules help:

• start with short, concrete nouns
• avoid vague multi-object phrases
• avoid over-describing the object unless needed
• try synonyms if the first prompt underperforms
• if possible, test singular and plural forms only after trying the simplest base term

For example, soldering iron is likely a better prompt than small metal electronics repair tool with handle. The latter contains more words, but that does not automatically make it better.

You should also expect some prompts to fail for perfectly normal reasons:

• the object may be tiny
• the prompt may be semantically broad
• the object may be heavily occluded
• the prompt may describe a category the model has weak visual grounding for

This is simply the cost of asking a flexible model to generalize beyond a fixed short class list.

Good Prompts vs. Weak Prompts

A few concrete examples make this easier to internalize:

• soldering iron: stronger than tool
• traffic light: stronger than street object
• barcode scanner: stronger than electronics device
• bus: stronger than large road vehicle with windows

The pattern is consistent. Good prompts are usually short, concrete, and visually grounded. Weak prompts are often too broad, too abstract, or too wordy. If the first prompt underperforms, try a nearby synonym before you conclude the model cannot handle the category at all.

Reading the Result, Not Just Looking At It

Annotated images are useful, but the result object tells you more than the picture does. Printing the top class names and confidence values makes it easier to inspect the output programmatically.

Open-vocabulary work often involves a short loop:

• try a prompt
• inspect detections
• revise the prompt
• re-run inference

This feedback loop is why an interactive workflow works well for Lesson 1. You are not yet building a packaged application. You are exploring the detector’s behavior iteratively.

Inspecting the Result Object

If you want to move one step beyond screenshots, inspect the prediction object directly:

result = text_results[0]

print(result.names)
print(result.boxes.xyxy[:3])
print(result.boxes.conf[:3])
print(result.boxes.cls[:3])

if result.masks is not None:
    print(result.masks.data.shape)

That quick inspection tells you almost everything you need for downstream work:

• result.names: maps class identifiers (IDs) to label names
• result.boxes.xyxy: stores box coordinates
• result.boxes.conf: stores confidence scores
• result.boxes.cls: stores predicted class IDs
• result.masks: appears because we are using segmentation-first checkpoints, so mask tensors are available alongside the boxes

This is the easiest bridge from “nice demo” to “usable building block.” Once readers understand where the predictions live, they can start saving detections, filtering them, or passing them into a larger pipeline.

Why Visual Prompting Is the Real Superpower

Text prompts are the easiest entry point, but visual prompting is where YOLOE-26 starts to feel genuinely different from a normal detector.

Sometimes words are not enough. Maybe the object is a very specific industrial part. Maybe the text label is ambiguous. Maybe the thing you want to find is easier to show than describe.

That is what visual prompting is for.

How Visual Prompts Are Structured

With visual prompts, you give the model one or more bounding boxes around reference objects. YOLOE-26 then uses those examples to find visually similar instances.

The next example uses the exact same bus.jpg sample image from the docs. To keep the first visual-prompt workflow stable and easy to reproduce, we place 1 prompt box around a person.

import numpy as np

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor
from ultralytics.utils import ASSETS

visual_model = YOLOE("yoloe-26s-seg.pt")
visual_model.set_classes(["person"])
visual_prompts = dict(
   bboxes=np.array(
       [
           [221.52, 405.8, 344.98, 857.54],
       ]
   ),
   cls=np.array([0]),
)

visual_result = visual_model.predict(
   ASSETS / "bus.jpg",
   visual_prompts=visual_prompts,
   predictor=YOLOEVPSegPredictor,
   conf=0.25,
)[0]
visual_result.names = {0: "person"}
visual_result.show()

The only slightly unusual part here is the visual_prompts dictionary:

• bboxes: contains the reference box
• cls: contains a sequential ID that associates that box with the prompt category

These are not global COCO class IDs. They are temporary identifiers for the prompt session.

For visualization, we also relabel that temporary prompt ID so the plotted output says person instead of object0.

This is the mode many tutorials skip, but it is extremely practical. If you can show the model what an object looks like once, you can turn that into a lightweight one-shot retrieval workflow.

When Visual Prompting Beats Text Prompting

Visual prompting is most useful when the object is easier to show than to describe. Suppose you are looking for a very specific wrench, connector, or control knob. A text label like metal connector may be too broad, but a carefully drawn reference box can tell the model, “Find more objects that look like this.” In practice, make the prompt box tight, representative, and light on background clutter. Very small prompt regions can be brittle, so start with larger, visually distinctive examples when you are learning the workflow.

Optional Extension: Prompting from a Separate Reference Image

One optional demo is also worth mentioning: the prompt does not have to come from the same image as the target. The refer_image argument handles that case. This is closer to a real retrieval workflow, where you have one known example and want to search a different frame set or batch for similar objects.

If you want to include this extension, here is a minimal working example:

import numpy as np

from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor
from ultralytics.utils import ASSETS

reference_model = YOLOE("yoloe-26s-seg.pt")
reference_model.set_classes(["person"])

reference_prompts = dict(
   bboxes=np.array([[221.52, 405.8, 344.98, 857.54]]),
   cls=np.array([0]),
)

reference_result = reference_model.predict(
   ASSETS / "zidane.jpg",
   refer_image=ASSETS / "bus.jpg",
   visual_prompts=reference_prompts,
   predictor=YOLOEVPSegPredictor,
   conf=0.10,
)[0]

reference_result.names = {0: "person"}
reference_result.show()

Treat this as an advanced extension, not a required part of the first pass through the lesson.

YOLOE-26 Prompt-Free Open-Vocabulary Object Detection

YOLOE-26 also has prompt-free variants. These models come with a built-in open vocabulary and do not require your own text prompts or visual prompts at inference time.

How Prompt-Free Inference Works in Practice

Here is the corresponding code:

from ultralytics import YOLOE
from ultralytics.utils import ASSETS

prompt_free_model = YOLOE("yoloe-26s-seg-pf.pt")
prompt_free_results = prompt_free_model.predict(ASSETS / "bus.jpg", conf=0.25)
prompt_free_results[0].show()

This mode is useful when you want a larger built-in vocabulary with minimal setup. According to the Ultralytics documentation, the prompt-free models use a built-in large vocabulary and internal embeddings for open-set recognition.

But there is a tradeoff, and it is important enough to say clearly: prompt-free convenience costs accuracy.

Prompt-free mode is appealing because it feels effortless. Load the model, run inference, and get a wide-vocabulary result without deciding on prompts first. In practice, it is best for exploratory analysis and quick discovery passes, not for the cases where you need the strongest possible precision on one narrow concept. If text prompting is you saying “look for this,” prompt-free mode is closer to saying “show me what the built-in vocabulary thinks is here.”

Where YOLOE-26 Beats YOLO26, and Where It Does Not

YOLOE-26 beats standard YOLO26 whenever the target category is dynamic, long-tail, or simply outside the fixed closed-set label space. If your environment changes often, or if you need to search for unusual objects without retraining a detector from scratch, YOLOE-26 is the more flexible tool.

It also has a strong headline benchmark story. Ultralytics reports that the x model reaches:

• 40.6 AP: on LVIS minival with text prompts
• 38.5 AP: with visual prompts
• 31.1 AP: in the prompt-free non-E2E setting

That is a meaningful spread. Prompt-free mode is convenient, but you pay for that convenience in accuracy.

The pattern is straightforward: the more explicit guidance you give YOLOE-26, the better it performs. Prompt-free mode is still useful, but it should be treated as a convenience mode, not the default path when accuracy matters most.

YOLO26 and YOLOE-26 at a Glance

A Practical Decision Framework

At the same time, YOLOE-26 is not automatically “better” than YOLO26 for every production workload. If your class list is fixed, your latency budget is tight, and you already know exactly what you need to detect, a standard fine-tuned YOLO26 model is often the simpler production artifact.

Here is a clean decision framework:

• YOLO26: choose when the classes are fixed and you care most about a lean production artifact
• YOLOE-26 text prompting: choose when you know the concept but want the flexibility to change classes without retraining
• YOLOE-26 visual prompting: choose when the object is easier to show than to describe
• YOLOE-26 prompt-free: choose when you want broad exploratory coverage and are willing to trade accuracy for convenience

This decision framework is exactly why Lesson 1 comes before Lesson 2. You need to understand where YOLOE-26 shines before you can use it intelligently as a labeling engine later.

Common Failure Modes and How to Debug Them

Open-vocabulary detection is powerful, but it is not magic. Here are the most common reasons your first run may disappoint you:

The Prompt Is Too Broad

A prompt (e.g., tool or electronics device) may be semantically valid but visually broad. The model has too many ways to satisfy the prompt, so detections may become noisy.

Fix: start narrower. Use the most concrete noun you can.

The Prompt Is Too Obscure

Sometimes the opposite happens. You use a very domain-specific term that the model has weak grounding for.

Fix: try a simpler synonym or a more common parent concept first.

The Object Is Too Small

Tiny objects are hard for almost every detector. Open-vocabulary capability does not remove that challenge.

Fix: use a larger input size, crop the image, or test on closer examples before judging the prompt.

The Visual Prompt Box Is Poor

If your reference box includes too much background or cuts off the object, you are teaching the model the wrong visual concept.

Fix: redraw the prompt box tightly and try again.

You Are Expecting Prompt-Free Mode to Behave Like Curated Prompting

Prompt-free is convenient, not optimal. If the results are underwhelming, that does not mean YOLOE-26 as a whole is weak. It may simply mean you should move to text or visual prompting.

Fix: switch to a more controlled prompt mode before drawing conclusions.

Your Prompts Overlap Semantically

Some prompt sets are too close to each other. If you ask for overlapping concepts (e.g., person, pedestrian, and worker, or tool, hand tool, and screwdriver), the detector may produce messy or merged behavior because multiple prompts can plausibly match the same region.

Fix: start with clearly separated categories. Add finer-grained prompts only after the coarse categories are behaving the way you expect.

The right question is not “Did one random prompt work perfectly on the first try?” It is “How much controllability do we get, and how quickly can we improve the result through prompting?”

One Deployment Detail You Should Not Miss

One operational detail from the Ultralytics docs deserves explicit attention. When you export a YOLOE model, the configured classes are baked into the exported weights. After that, you cannot keep swapping prompt classes on the exported artifact. To change them, you need to re-export from the original checkpoint.

During experimentation, this is not a big deal. You just change the prompts and rerun the code. But if you are preparing a deployable artifact, the prompt configuration becomes part of what you are exporting.

This is one of the clearest ways to understand the line between experimentation and production:

• exploration mode: YOLOE-26 is highly flexible
• deployment mode: some of that flexibility gets frozen into the exported artifact

That detail will matter a lot in Lesson 2, because Lesson 2 is about what happens when you stop exploring and start converging on a narrower detector workflow.

What's next? We recommend PyImageSearch University.

Course information:
120+ total classes • 115+ hours of on-demand code walkthrough videos • Last updated: August 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 120+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 94+ Certificates of Completion
• ✓ 115+ hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

The easiest way to think about this lesson is:

• YOLO26 is your fast closed-set baseline.
• YOLOE introduced promptable open-vocabulary detection to the YOLO family.
• YOLOE-26 brings that same idea into the YOLO26 generation with stronger open-vocabulary performance and the same real-time mindset.

By now, you have seen 3 concrete workflows:

• Text prompting: inference with set_classes()
• Visual prompting: inference with reference boxes
• Prompt-free inference: inference with a built-in vocabulary

You have also seen the deeper point behind those workflows. YOLOE-26 changes how quickly we can move from “we have an object in mind” to “we have a detector doing something useful.”

That is already enough to start experimenting. But it also raises the next practical question.

What if you do not want a permanently flexible open-vocabulary detector? What if you want to use YOLOE-26 to discover and pre-label a niche category, then turn that into a small, fast, deployable closed-set detector?

That is exactly what we will do in Lesson 2.

Citation Information

Singh, V. “YOLO26 Open-Vocabulary Object Detection with YOLOE-26,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/mzpx3

@incollection{Singh_2026_yolo26-open-vocabulary-object-detection-yoloe-26,
  author = {Vikram Singh},
  title = {{YOLO26 Open-Vocabulary Object Detection with YOLOE-26}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/mzpx3},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post YOLO26 Open-Vocabulary Object Detection with YOLOE-26 appeared first on PyImageSearch.

Make a Chrome Extension to Digest Webpages with Manifest V3 and Groq API

In this lesson, you will learn how to build a Chrome extension that can read the current webpage, send that content to a large language model (LLM), and stream the answer back into a popup interface.

To learn how to build a Chrome extension that summarizes webpages with Manifest V3 and the Groq API, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

This is exactly the kind of project we want for a first lesson. The codebase is small enough to understand in one sitting, but practical enough to expose the moving parts that actually matter in a real browser-based artificial intelligence (AI) tool:

• Manifest V3 configuration
• Extension permissions
• Popup user interface (UI) structure
• Persistent storage with chrome.storage.local
• Reading live page content with chrome.scripting.executeScript
• Calling the Groq API with fetch
• Streaming tokens back into the interface
• Managing conversation history across turns

By the end of this lesson, you will understand what each file does, how the pieces fit together, and why this design works so well for a first Chrome LLM extension.

Meet the Project

Our project is called Page Intelligence. When the user clicks the extension icon, Chrome opens a popup. From that popup, the user can:

• Save a Groq application programming interface (API) key
• Summarize the current webpage
• Ask follow-up questions about that page
• Clear the current conversation and start over

At first glance, that workflow feels straightforward. Under the hood, though, it touches 3 separate environments:

• The popup UI, rendered by Chrome as an extension page
• The active tab, which contains the webpage we want to read
• The Groq API, which generates the response

That separation is the key architectural idea in this lesson. The popup cannot directly read the page Document Object Model (DOM) just because both are open in the same browser window. Instead, it has to ask Chrome for permission and inject code into the active tab at the right moment.

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Configuring Your Development Environment

To follow this lesson, you only need Google Chrome, a Groq API key, and the project files for the extension. Since this is a lightweight Chrome extension project, there are no Python packages or JavaScript dependencies to install.

Then follow these steps:

• Download or open the project folder.
• Get a Groq API key.
• Open chrome://extensions.
• Enable Developer mode.
• Click Load unpacked and select the project folder.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Project Structure

We first need to review our project directory structure.

Start by accessing this tutorial’s “Downloads” section to retrieve the source code.

From there, take a look at the directory structure:

chrome-llm-extension/
├── manifest.json
├── popup.html
├── popup.js
├── styles.css
└── README.md

The minimal layout is intentional. It keeps the lesson focused. There is no framework, no extra build system, and no separate content script file to chase through. All of the user-facing logic lives in the popup, and the extension reaches into the current page only when it needs to read content.

What Each File Is Responsible For

manifest.json

• Declares that this is a Manifest V3 extension
• Requests the permissions needed to read the active tab, inject a script, and store the API key
• Registers popup.html as the default popup

popup.html

• Defines the structure of the popup interface
• Creates the API key panel, chat area, quick action buttons, and message box

styles.css

• Controls layout, spacing, colors, bubble styles, and the overall feel of the popup

popup.js

• Contains the extension’s runtime logic
• Loads and stores the API key
• Reads page content from the active tab
• Calls the Groq API
• Streams the response token by token
• Maintains chatHistory

Why This Structure Works Well for a Lesson

For a first Chrome extension project, fewer files usually mean faster understanding. Instead of bouncing between a popup, a background worker, and multiple content scripts, you get to follow one direct path:

• Chrome opens the popup.
• The popup loads saved state.
• The popup requests page text when needed.
• The popup sends a streaming LLM request.
• The popup updates the UI in real time.

For a teaching project, that is a great tradeoff. We stay practical without burying the learner in scaffolding.

The Chrome Extension Mental Model

Before we go any further, let us make the execution model crystal clear.

A popup is not the webpage

When you click a Chrome extension icon, Chrome opens a small Hypertext Markup Language (HTML) page that belongs to the extension. That page has its own DOM, JavaScript context, and permissions.

This means:

• popup.js can manipulate elements in popup.html
• popup.js cannot directly access document.body from the currently open website
• To read the website, the extension must ask Chrome to run code inside the active tab

This is exactly why chrome.scripting.executeScript matters in this project.

Storage Belongs to the Extension, Not the Site

There is a second separation we need to keep in mind, and that is storage.

If you used localStorage inside a normal webpage, that data would belong to that page’s origin. Here, we want the API key to belong to the extension itself, regardless of which site the user is visiting.

For that reason, the code uses:

chrome.storage.local

This storage area is managed by Chrome and shared across extension contexts.

Network Requests Must Be Explicitly Allowed

Extensions are permission-driven. If you want to send requests to the Groq API, you declare that in manifest.json using host_permissions.

This is one of the core Chrome extension design principles:

• UI and code live inside the extension
• Privileges are declared in the manifest
• Cross-context access is granted only through the proper APIs

Walking Through manifest.json

Open manifest.json, and you will find the contract between your extension and Chrome.

Here is the full file:

{
  "manifest_version": 3,
  "name": "Page Intelligence",
  "version": "1.0",
  "description": "Chat with any webpage using an LLM. Built for the Agent AI course.",
  "permissions": ["activeTab", "scripting", "storage"],
  "host_permissions": ["https://api.groq.com/*"],
  "action": {
    "default_popup": "popup.html",
    "default_title": "Page Intelligence"
  }
}

manifest_version: 3

Chrome expects modern extensions to use Manifest V3. That affects how extensions are structured, how scripts are injected, and how background logic is handled.

In this project, Manifest V3 gives us a clean, current, production-relevant starting point.

Why These Permissions Matter

activeTab

This permission allows the extension to interact with the tab the user is currently using. In a tool like this, that is essential because the extension must read the active page when the user clicks Summarize Page.

scripting

This permission unlocks chrome.scripting.executeScript, which is the modern Manifest V3 (MV3) way to inject code into a page.

storage

This permission lets the extension persist the Groq API key using chrome.storage.local.

Why host_permissions Matters

The extension calls https://api.groq.com/openai/v1/chat/completions.

Without the proper host permission, that request would be blocked. This line gives the extension permission to communicate with the Groq API host.

Why action.default_popup Matters

Here we define the entry point for the user experience. This tells Chrome, “when the extension icon is clicked, open popup.html.”

One line is all it takes to wire the entire interface into the browser.

Understanding popup.html

Next, open popup.html. If manifest.json is the contract, popup.html is the stage where the user interacts with the extension.

The Popup Is Intentionally Split into Clear UI Regions

The HTML defines 5 major regions:

• Header
• API key section
• Chat history
• Quick actions
• Input area

That separation matters because each region maps directly to behavior in popup.js.

Here is the structural core of the file:

<div class="container">
  <div class="header">...</div>
  <div id="api-key-section" class="api-key-section">...</div>
  <div id="chat-history" class="chat-history">...</div>
  <div class="quick-actions">...</div>
  <div class="input-area">...</div>
</div>

The Header Sets Up Identity and Settings Access

At the top, we have:

<h1>Page Intelligence</h1>
<button id="settings-btn" class="icon-btn">...</button>

The lesson here is simple but important. Even a tiny extension UI needs clear affordances. The settings button gives the user a predictable place to reopen the API key panel after the initial save.

The API Key Section Introduces Stateful UI

The API key input exists in the DOM from the beginning, but JavaScript decides whether it should be visible based on whether a key has already been saved.

A common frontend pattern is at work here:

• Keep the structure in HTML
• Let JavaScript decide the current state

The Chat Area Is the Main Feedback Surface

The #chat-history container starts with a welcome message:

<div class="welcome-msg">
  Ask me anything about this page,<br />
  or click <strong>Summarize</strong> to get started.
</div>

This does more than decorate the popup. It solves the empty-state problem. Before the first interaction, the popup still feels guided and purposeful.

Quick Actions Reduce Friction

The 2 action buttons are:

• Summarize Page
• Clear

These are excellent teaching examples because they show 2 different UI intents:

• Task button: launches a multi-step workflow
• Reset button: clears local state

The Input Area Supports Chat-Like Behavior

The textarea plus send button give the user a second interaction path. They can either:

• Click the guided summary flow first
• Skip directly to a custom question

Practically, this supports both beginners and more confident users.

Understanding styles.css

Now open styles.css. This is where the extension begins to feel polished instead of merely functional.

The Layout Uses a Simple But Effective Flex Column

The 2 most important layout rules are:

body {
  width: 400px;
}

.container {
  display: flex;
  flex-direction: column;
  height: 560px;
}

This gives the popup a fixed footprint and lets the chat area grow while the header, actions, and input stay anchored.

Why the Chat Panel Works Well

The chat container uses:

.chat-history {
  flex: 1;
  overflow-y: auto;
  display: flex;
  flex-direction: column;
  gap: 10px;
}

You will see this pattern often in messaging interfaces:

• flex: 1: lets the panel absorb remaining height
• overflow-y: auto: makes long conversations scrollable
• gap: keeps messages visually separated without hard-to-maintain margins

Message Bubbles Encode Role Visually

The Cascading Style Sheets (CSS) differentiate:

• .message.user
• .message.assistant
• .message.error
• .message.system

This is not just cosmetic. Good styling teaches the user how to read the conversation. With one glance, they can tell which text came from them, which came from the model, and whether a message is informational or an error.

Small Animation Helps the Interface Feel Alive

The fadeIn keyframe is subtle:

@keyframes fadeIn {
  from { opacity: 0; transform: translateY(4px); }
  to   { opacity: 1; transform: translateY(0); }
}

It is also a good reminder that user experience (UX) is part of engineering. Because the model response streams in incrementally, the interface benefits from motion that feels lightweight and responsive.

Input and Actions Are Styled for Fast Iteration

The quick action buttons and textarea focus states give the extension a clean, modern feel without pulling in a UI framework.

This leads to another useful lesson from the project:

• HTML: handles structure
• CSS: handles presentation
• JavaScript: handles behavior

For a beginner-friendly extension, that separation is exactly what we want.

Reading README.md the Right Way

It is easy to ignore README.md, but in a teaching repository, that file matters.

The README Is the Learner’s Runway

This project’s README does 3 useful jobs:

• Explains: what concepts the extension teaches
• Shows: setup steps for getting a Groq API key and loading the extension
• Presents: the codebase structure in a fast, approachable way

So while README.md is not runtime code, it is still part of the learning architecture of the project.

Why This Matters in Real Projects

When you teach or ship developer tooling, code alone is not enough. Learners need:

• Context
• Setup steps
• A map of what to read first

This repository already does a good job of that by aligning the README sections with the files learners will inspect next.

Walking Through popup.js

With the surrounding files covered, we can move to the heart of the extension.

popup.js is where the extension:

• Loads saved state
• Handles button clicks
• Reads the active page
• Calls the Groq API
• Streams the assistant output
• Updates the chat UI

This single file is small enough to follow in one sitting, which makes it ideal for a hands-on lesson.

Configuration and State Live at the Top

The file begins with configuration values:

const GROQ_API_URL = "https://api.groq.com/openai/v1/chat/completions";
const MODEL = "meta-llama/llama-4-scout-17b-16e-instruct";

Then it defines a reusable system prompt and 2 key pieces of runtime state:

let apiKey = "";
let chatHistory = [];

This pattern is worth teaching because it keeps the mental model clean:

• Constants describe how the app talks to the model
• Mutable state tracks what the user has done in this popup session

DOM References Create a Bridge from HTML to Behavior

The next block grabs the key UI elements:

const apiKeySection = document.getElementById("api-key-section");
const apiKeyInput   = document.getElementById("api-key-input");
const saveKeyBtn    = document.getElementById("save-key-btn");
const settingsBtn   = document.getElementById("settings-btn");
const chatHistoryEl = document.getElementById("chat-history");
const summarizeBtn  = document.getElementById("summarize-btn");
const clearBtn      = document.getElementById("clear-btn");
const userInput     = document.getElementById("user-input");
const sendBtn       = document.getElementById("send-btn");

This is why the element identifiers (IDs) in popup.html matter. They are the handles that let JavaScript attach behavior to markup.

DOMContentLoaded Restores Saved Extension State

When the popup opens, this listener runs:

document.addEventListener("DOMContentLoaded", async () => {
  const stored = await chrome.storage.local.get("groq_api_key");

  if (stored.groq_api_key) {
    apiKey = stored.groq_api_key;
    apiKeySection.style.display = "none";
  }
});

Here is one of the first places where theory and practice meet:

• The popup is ephemeral. It opens when the user clicks the extension icon.
• Any in-memory JavaScript state disappears when the popup closes.
• Persistent state must therefore live outside regular variables.

This is exactly why chrome.storage.local is essential here. It gives us persistence without forcing us to introduce a database or a heavier architecture.

Saving the API Key

When the user clicks Save, the extension:

• Reads the input value
• Validates that it is not empty
• Stores it with chrome.storage.local.set(...)
• Copies it into the in-memory apiKey variable
• Hides the API key panel
• Shows a confirmation message

What you are seeing is a tidy example of state synchronization between:

• The DOM
• Extension storage
• In-memory JavaScript state

The Settings Button Reopens the Key Panel

This small handler is worth pointing out:

settingsBtn.addEventListener("click", () => {
  apiKeySection.style.display =
    apiKeySection.style.display === "none" ? "flex" : "none";
});

Again, not every useful behavior needs a framework. For a focused popup, direct DOM manipulation is perfectly reasonable and easy to understand.

How the Extension Reads the Current Webpage

This is the most important architectural jump in the lesson, so take a moment to make sure the flow is clear before moving on.

getPageText() Starts by Locating the Active Tab

The function begins with:

const [tab] = await chrome.tabs.query({ active: true, currentWindow: true });

In plain English, this asks Chrome to return the tab the user is looking at right now.

The Extension Then Injects a Function into That Tab

Here is the core pattern:

const results = await chrome.scripting.executeScript({
  target: { tabId: tab.id },
  func: () => {
    const clone = document.body.cloneNode(true);
    clone.querySelectorAll("script, style, nav, footer, aside")
      .forEach(el => el.remove());

    return {
      title: document.title,
      url: window.location.href,
      text: clone.innerText.replace(/\s+/g, " ").trim().slice(0, 8000),
    };
  },
});

As a teaching example, this snippet is excellent because it shows the exact boundary between extension code and page code.

Why Clone the Page Body First

The code uses:

const clone = document.body.cloneNode(true);

Cloning first is smart because it lets the extension clean the content without altering the actual webpage the user is viewing.

Why Remove Script, Style, Nav, Footer, and Aside

This is a lightweight content-cleaning step. The extension wants the main readable text, not the surrounding noise.

This gives learners a practical introduction to preprocessing before sending data to an LLM.

Why Normalize Whitespace and Cap Text Length

This line matters:

clone.innerText.replace(/\s+/g, " ").trim().slice(0, 8000)

It does 3 useful things:

• Collapses repeated whitespace
• Trims leading and trailing space
• Limits the payload to 8,000 characters

The final limit is especially important. Even though modern models have large context windows, good engineering still means keeping inputs focused and predictable.

The Return Value Comes Back to the Popup

After executeScript finishes, the popup receives:

• The page title
• The page uniform resource locator (URL)
• Cleaned text content

This makes getPageText() the bridge from the browser page to an LLM-ready prompt.

How the Groq Streaming Call Works

Once the extension has page content, it needs to send a chat completion request and render the answer in real time.

streamResponse() Checks for the API Key First

The function opens with a guard clause:

if (!apiKey) {
  apiKeySection.style.display = "flex";
  showSystem("Please set your Groq API key first.", "error");
  return;
}

This is good defensive programming. Before the app makes a network request, it verifies that the minimum required state is present.

The Request Body Sends the Full Conversation

The fetch call includes:

body: JSON.stringify({
  model: MODEL,
  messages: chatHistory,
  stream: true,
  temperature: 0.7,
  max_tokens: 1024,
})

There are 2 especially important details here.

messages: chatHistory

This is how the model gets context. Instead of sending only the newest user message, the extension sends the full running conversation.

As a result, the model can handle:

• Follow-up questions
• Clarifications
• Multi-turn interaction grounded in prior turns

stream: true

This tells the API to return response content instead of waiting for the full answer to be finished.

That is what gives the extension its chat-like feel.

The Response Is Read as a Stream

The code then creates:

const reader  = response.body.getReader();
const decoder = new TextDecoder();

This is where the frontend side starts to feel more advanced. Instead of calling await response.json(), the extension reads raw chunks from the response body.

Server-Sent Events (SSE) Are Parsed Line by Line

The code looks for lines that begin with: data:

It then trims the prefix, checks for [DONE], and parses the JavaScript Object Notation (JSON) payload:

const parsed = JSON.parse(payload);
const token  = parsed.choices?.[0]?.delta?.content ?? "";

If content is present, the function sends it to onChunk(token).

Why the UI Feels Responsive

Every arriving content chunk updates the assistant bubble immediately. That means the user does not stare at a frozen popup waiting for a full paragraph to appear all at once.

There is a practical UX lesson here:

• Streaming improves perceived speed
• Real-time rendering makes the app feel more interactive
• Even a simple UI can feel polished if feedback arrives continuously

One Subtle Engineering Note

This implementation splits each decoded chunk by newline and skips JSON parsing errors when a partial fragment arrives at a chunk boundary.

For a teaching demo, that tradeoff is acceptable. In a more robust production version, you would usually maintain a rolling buffer so partial SSE lines can be reconstructed correctly before parsing.

That is a valuable lesson too: simple implementations are often ideal for learning, even when they are not yet industrial-strength.

How Conversation State Is Managed

A good chat interface is really a state-management problem wearing a friendly UI.

sendMessage() Handles the Main Interaction Loop

At a high level, sendMessage() does the following:

• Cleans the user input
• Pushes the user message into chatHistory
• Renders the user bubble
• Disables the input while the model responds
• Creates an empty assistant bubble
• Streams response content into that bubble
• Saves the final assistant response back into chatHistory
• Re-enables the input

This is the core interaction cycle of the whole extension.

Why Create an Empty Assistant Bubble First

This line is the trick:

const assistantBubble = appendMessage("assistant", "");

Instead of waiting for the final answer, the UI creates a placeholder bubble and fills it as response content arrive. That is what makes streaming visible to the user.

Auto-Scroll Keeps the Newest Response Content in View

During streaming, the code does:

chatHistoryEl.scrollTop = chatHistoryEl.scrollHeight;

This small line is easy to miss, but it matters. Without it, long responses would stream out of view and the interface would feel clumsy.

The System Prompt Is Injected When Needed

When the user clicks Send on a fresh chat, the code makes sure the system prompt is present:

if (chatHistory.length === 0) {
  chatHistory.push({ role: "system", content: SYSTEM_PROMPT });
}

This is a compact but important pattern. It ensures that every conversation begins with the assistant’s behavioral instructions, even if the user skips the summarize flow and goes straight to a custom question.

The Enter Key Is Tuned for Chat UX

This handler:

if (e.key === "Enter" && !e.shiftKey) {
  e.preventDefault();
  sendBtn.click();
}

gives the interface a familiar messaging behavior:

• Enter sends
• Shift + Enter can still be used for a multiline prompt

Small details like this make the project feel thoughtful.

How the Summarize Workflow Works

The Summarize Page button is where all the major ideas in the project come together.

The Button First Updates Its Own UI State

When clicked, it temporarily changes to: "Reading page..."

This gives the user immediate feedback that work has started.

The History Is Reset for a Page-First Conversation

Inside the click handler, the code sets:

chatHistory = [
  { role: "system", content: SYSTEM_PROMPT },
];

This is a deliberate design decision. The summarize flow creates a fresh conversation centered on the current page.

The Page Content Is Converted into a User Message

The extension builds a message like this:

const userMessage =
  `Please summarise the following webpage.\n\n` +
  `Title: ${page.title}\nURL: ${page.url}\n\nContent:\n${page.text}`;

This is a clever and practical technique. Instead of inventing a separate prompt schema, the extension simply turns page context into a normal user message. That means the rest of the chat pipeline can stay unchanged.

Why This Design Is Elegant

The same sendMessage() function handles:

• Free-form user questions
• A page summary request

That reduces duplicated logic and keeps the codebase teachable.

How the Clear Button Resets the Interface

The clear handler is short, but it teaches an important UI principle.

Reset Both State and Visible Output

When the user clicks Clear, the code:

• Empties chatHistory
• Restores the welcome message HTML inside #chat-history

This gives the user a clean slate both logically and visually.

In many small projects, bugs appear because developers reset one layer but forget the other. This extension avoids that trap by resetting both.

End-to-End Flow, From Click to Answer

Let us put the whole lesson together in one concrete sequence.

What Happens When the User Clicks Summarize Page

• Chrome opens the extension popup.
• popup.js loads and restores any saved API key from chrome.storage.local.
• The user clicks Summarize Page.
• getPageText() asks Chrome for the active tab.
• Chrome injects a function into that page using chrome.scripting.executeScript.
• The injected function clones the page body, removes noisy elements, and returns title, URL, and cleaned text.
• The popup resets chatHistory with the system prompt.
• The popup creates a user message containing the page content.
• sendMessage() renders the user bubble and creates an empty assistant bubble.
• streamResponse() sends the chat request to Groq with stream: true.
• The popup reads SSE chunks, extracts response content, and updates the assistant bubble live.
• When streaming ends, the full assistant message is saved into chatHistory.

That is the full extension lifecycle in one pass.

What Happens on the Next Follow-Up Question

The second turn is even more interesting:

• The user asks a follow-up question in the textarea.
• sendMessage() appends that new message to the existing chatHistory.
• The full conversation is sent again to the model.
• The model now has access to both the page context and the prior summary.

That is what makes the extension feel conversational instead of one-shot.

Practical Engineering Takeaways

This extension may be small, but it teaches several patterns that show up in larger AI products too.

Pattern 1: Retrieve First, Generate Second

The extension does not ask the model to guess what is on the page. It retrieves the page content first, then sends that content to the model.

The retrieval step is simple, but it is foundational.

Pattern 2: Separate UI State from Persistent State

The API key lives in chrome.storage.local.

The active conversation lives in memory as chatHistory.

This is a healthy separation because:

• Long-lived secrets survive popup closures
• Short-lived conversations reset naturally when the session changes

Pattern 3: Stream Whenever Responsiveness Matters

Streaming is not just a flashy feature. It changes how the user experiences latency.

Even if the total response time stays similar, streaming makes the system feel faster and more alive.

Pattern 4: Keep the First Version Intentionally Small

This extension could have included:

• A background service worker
• Conversation persistence across popup sessions
• Rich markdown rendering
• Better page extraction heuristics
• More robust streaming buffering

But for a lesson, the current scope is exactly right. It is complete enough to be useful and small enough to fully understand.

Where You Could Take This Project Next

Once learners understand this version, there are several natural improvements worth exploring.

Improve Content Extraction

Right now, the extension removes a few noisy tags and then uses innerText.

A stronger version could:

• Prefer main or article-like containers when present
• Skip repeated sidebar content more aggressively
• Chunk long pages instead of truncating them at 8,000 characters

Improve Streaming Robustness

The current SSE parsing is easy to understand, which is excellent for a first lesson. A more production-oriented version could keep a line buffer across chunk boundaries before parsing JSON.

Improve Conversation Persistence

chatHistory currently lives only in memory. If you close the popup, the conversation disappears.

That is fine for a lesson, but a future version could store conversations in chrome.storage.local or IndexedDB.

Improve Prompting

The current system prompt is short and sensible. A later lesson could show how to:

• Ask for citation-style answers grounded in extracted text
• Detect when the page content is too thin
• Adapt response style for summarization versus question answering

What's next? We recommend PyImageSearch University.

Course information:
120+ total classes • 115+ hours of on-demand code walkthrough videos • Last updated: August 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 120+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 94+ Certificates of Completion
• ✓ 115+ hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this lesson, you saw how a compact Chrome extension can combine browser APIs and LLM APIs into a practical, real-time workflow.

You learned how:

• manifest.json: declares the extension’s privileges and popup entry point
• popup.html: structures the user interface
• styles.css: makes the UI readable and responsive
• README.md: supports learning and onboarding
• popup.js: ties everything together through storage, page extraction, streaming, and chat state

Most importantly, you learned the core architectural idea behind the project:

• read the current page with the proper Chrome API
• turn that content into model-ready context
• stream the answer back into a lightweight interface

This pattern is simple, useful, and widely applicable. Once you understand it here, you can reuse it in richer browser tools, agent workflows, and AI-powered productivity extensions.

Citation Information

Singh, V. “Make a Chrome Extension to Digest Webpages with Manifest V3 and Groq API,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/8q9l7

@incollection{Singh_2026_make-chrome-extension-digest-webpages-manifest-v3-groq-api,
  author = {Vikram Singh},
  title = {{Make a Chrome Extension to Digest Webpages with Manifest V3 and Groq API}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/8q9l7},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Make a Chrome Extension to Digest Webpages with Manifest V3 and Groq API appeared first on PyImageSearch.

Scaling, Optimizing, and Exporting Transformers with PyTorch Lightning

In this lesson, you will learn how to scale, optimize, and export Transformer models using PyTorch Lightning, from enabling mixed precision and distributed training to generating ONNX and TorchScript exports ready for production inference. You will see how Lightning’s configuration-driven design, combined with Hydra, lets you take the exact same training code from Lesson 1 and push it into a high-performance, deployment-friendly workflow.

This lesson is the last in a 2-part series on PyTorch Lightning:

1. Training with PyTorch Lightning: Structured MLOps Development
2. Scaling, Optimizing, and Exporting Transformers with PyTorch Lightning (this tutorial)

To learn how to scale, optimize, and export Transformer models with PyTorch Lightning, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Introduction to Scaling PyTorch Lightning Transformer Training

Scaling Transformer training and preparing models for real-world deployment typically requires complex engineering effort such as distributed training, mixed-precision optimization, and multiple export formats. But thanks to PyTorch Lightning and Hydra, you can achieve all of this without rewriting your codebase or adding low-level boilerplate. In this lesson, we will take the sentiment-classification project you built earlier and upgrade it into a fast, scalable, and deployment-ready workflow.

What We Built Previously

In the previous tutorial, you created a complete sentiment classification pipeline using DistilBERT.

You structured your project with:

• A LightningDataModule to handle dataset loading, tokenization, and data loaders
• A LightningModule containing the model, forward pass, loss, metrics, and optimizer
• A clean Hydra configuration hierarchy that controlled every part of the training run
• A flexible training script that handled seeding, callbacks, logging, and TensorBoard
• An inference script for single, batch, and interactive predictions

This foundation gave you a fully reproducible training workflow that runs the same way on CPU, GPU, or MPS, with settings that can be overridden easily using Hydra’s command-line syntax.

However, training a base model is only the first step in a production ML workflow.

Why Scaling and Exporting Matter

Real-world Transformer workloads often push beyond what a single GPU or even a single machine can handle.

Teams need:

• Mixed precision (FP16 or BF16): for faster training and lower memory usage
• Distributed data parallel training: to use multiple GPUs efficiently
• Gradient accumulation: to simulate large batch sizes
• Model exports (ONNX or TorchScript): for deployment in production systems
• Lightweight inference runtimes: faster than PyTorch eager mode
• Stable artifact folders: ready for DVC, CI/CD, or cloud deployment pipelines

These features turn a research-grade script into an MLOps-grade training system, and that is exactly what you will build in this lesson.

How Lightning Makes It Easy

The best part is that none of this requires rewriting your training loop.

PyTorch Lightning abstracts the hard parts:

• Mixed precision: uses a single configuration key
• Distributed training (DDP or FSDP): requires only a strategy change
• Export logic: can be cleanly added without modifying model code
• Hydra: lets you switch between configurations instantly
• Checkpoints, logs, and metrics: remain fully reproducible

Instead of modifying your DataModule or LightningModule, you will extend the trainer configuration and add a dedicated export script.

That is the power of Lightning: the same codebase now supports single-GPU training, multi-GPU scaling, and deployment-ready exports with minimal changes.

Would you like immediate access to 3,457 images curated and labeled with hand gestures to train, explore, and experiment with … for free? Head over to Roboflow and get a free account to grab these hand gesture images.

Configuring Your Development Environment

Lesson 2 introduces new capabilities (e.g., distributed training, mixed precision, and exporting models to ONNX or TorchScript), so the environment now includes additional dependencies specifically meant for scaling and production-grade inference.

Below is the exact requirements.txt used in this lesson:

# Lesson 2 Requirements
# Additional dependencies for production features

# PyTorch and Lightning
torch>=2.1.0
torchvision>=0.16.0
torchaudio>=2.1.0
pytorch-lightning>=2.1.0

# Hugging Face ecosystem
transformers>=4.35.0
datasets>=2.14.0
tokenizers>=0.15.0

# Configuration management
hydra-core>=1.3.0
omegaconf>=2.3.0

# Metrics and monitoring
torchmetrics>=1.2.0
tensorboard>=2.15.0

# Model export (LESSON 2 specific)
onnx>=1.15.0
onnxruntime>=1.16.0  # CPU version works on all platforms

# Data processing
numpy>=1.24.0
pandas>=2.0.0

# Utilities
tqdm>=4.66.0
pyyaml>=6.0.0

# Note: For NVIDIA GPUs on Linux, you can optionally install:
# onnxruntime-gpu>=1.16.0  # Requires CUDA, Linux only

​​This environment enables 3 major Lesson 2 features:

1. Multi-GPU Distributed Training

Powered by:

• PyTorch Lightning (DDP or FSDP strategies)
• Hydra configurations for trainer selection

2. Mixed Precision

torch >= 2.x unlocks CPU BF16, GPU AMP, and faster matrix kernels.

3. ONNX Export and Runtime Inference

The addition of onnx and onnxruntime enables:

• model export to ONNX format
• cross-platform CPU inference
• benchmarking ONNX vs PyTorch (optional)

If you are using an NVIDIA GPU on Linux, you can optionally install:

pip install onnxruntime-gpu

This is not required for the lesson (we stay framework-agnostic), but readers who want GPU ONNX inference can enable it easily.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Preparing PyTorch Lightning Models for Scalable Multi-GPU Training

Before we scale training, enable DDP or FSDP, or export models to ONNX or TorchScript, it is important that the project structure and configuration layout are set up correctly. Lesson 2 builds directly on the modular design from Lesson 1, but introduces specialized trainer configs and improved environment setup that unlock multi-GPU training and optimized inference.

Your updated project structure now reflects these goals.

Project Structure Refresher (Updated for Lesson 2)

Below is the exact directory layout used in this lesson:

.
├── configs
│   ├── config.yaml
│   ├── data
│   │   └── imdb.yaml
│   ├── model
│   │   └── distilbert.yaml
│   └── trainer
│       ├── ddp.yaml
│       ├── fsdp.yaml
│       └── single_gpu.yaml
├── README.md
├── requirements.txt
├── RUN.md
├── sample_reviews.txt
└── src
    ├── data_module.py
    ├── inference.py
    ├── model_module.py
    └── train.py

This layout introduces 3 major upgrades compared to Lesson 1:

Dedicated Trainer Configurations

You now have separate Hydra configurations under configs/trainer/ for:

• single_gpu.yaml: standard 1-GPU or CPU training
• ddp.yaml: multi-GPU Distributed Data Parallel (DDP)
• fsdp.yaml: Fully Sharded Data Parallel (memory-efficient large-model training)

This keeps the scaling logic completely outside the Python code (a major MLOps advantage).

Unified Root Configuration (config.yaml)

The root configuration composes the model, data, and trainer settings into a single experiment specification.

Switching between single-GPU and DDP training is as simple as:

python src/train.py trainer=ddp

No code changes or additional flags are required. By default, the project uses the DDP trainer (trainer=ddp), but you can switch to single-GPU training by running python src/train.py trainer=single_gpu.

Clean Separation of Code Files

Your src/ folder remains identical to Lesson 1 (data_module.py, model_module.py, train.py, and inference.py). This reinforces the Lesson 2 philosophy:

“Scaling and exporting should not require modifying your model code.”

Only the configs/ files change, not the implementation.

Hydra Enhancements for Scaling

Lesson 2 is where Hydra truly shines.

Instead of hardcoding distributed training logic in Python, you now have clean trainer profiles:

configs/trainer/
│── single_gpu.yaml
│── ddp.yaml
└── fsdp.yaml

Each YAML file defines:

• accelerator
• devices
• strategy (ddp or fsdp)
• precision
• logging settings

Examples a reader will use later:

python src/train.py trainer=single_gpu
python src/train.py trainer=ddp
python src/train.py trainer=fsdp

Hydra swaps in the right configuration automatically.

This is exactly what you expect in a real MLOps pipeline.

Why This Setup Matters (Before We Scale)

This lightweight structure unlocks all the heavy features coming next:

• Training: can scale from CPU to GPU to multi-GPU without touching code
• Export pipelines (ONNX or TorchScript): work consistently because configs capture preprocessing
• FSDP and mixed precision: require stable config-driven Trainer settings
• Model reproducibility: is guaranteed across environments

Most importantly:

You now have a production-grade pattern:

• Model logic: stays in model_module.py
• Data logic: stays in data_module.py
• Training logic: stays in train.py
• Scaling and exporting logic: stays in configs/*

This is the exact separation used in serious MLOps workflows.

Revisiting the Code Architecture

Before we introduce mixed precision, distributed training, gradient accumulation, and model export, it is important to ground ourselves in the architecture we built in the previous tutorial. Lesson 1 gave us a clean, modular foundation. Lesson 2 builds directly on this foundation without changing most of the code. That is the real strength of PyTorch Lightning and Hydra: scaling does not require rewriting your project.

DataModule, LightningModule, and Hydra Recap

Our project continues to rely on the same 3 core abstractions:

LightningDataModule

Handles all data concerns:

• Dataset: downloading the IMDB dataset
• Tokenization: using a Hugging Face tokenizer
• Data loaders: creating training, validation, and test DataLoaders
• Separation of concerns: keeping preprocessing separate from training logic

This structure remains untouched in Lesson 2. AMP, DDP, and export do not require modifying data_module.py.

LightningModule

Encapsulates the model architecture and training logic:

• Encoder: DistilBERT
• Forward pass: processing model inputs
• Loss and metrics: computing training loss and evaluation metrics
• Optimizer: configuring optimization

Lesson 2 also does not modify the internal model logic.

Instead, we scale training through the Trainer and Hydra configuration.

Hydra

Hydra remains the central controller of the entire workflow.

It composes the configuration from:

• configs/model/*.yaml
• configs/data/*.yaml
• configs/trainer/*.yaml

Lesson 2 extends Hydra with:

• DDP trainer configurations
• FSDP configurations (if desired)
• AMP configurations
• gradient accumulation
• export configurations

But the training code (train.py) stays nearly the same.

This consistent architecture is what makes the next steps (i.e., scaling, exporting, and optimizing) feel incremental rather than overwhelming.

Preparing for Advanced Training

Lesson 2 introduces new capabilities commonly needed in real-world pipelines:

• multi-GPU distributed training
• mixed precision training
• larger effective batch sizes
• ONNX or TorchScript model export
• ONNX Runtime production inference

Here is the key design rule we follow:

We scale the system without touching the model or data code unless absolutely necessary.

Lightning was built exactly for this: features (e.g., AMP and DDP) are injected through the Trainer, not through the model.

Hydra was built to help you swap configurations without modifying Python files. This keeps your codebase stable (a critical requirement for teams and MLOps pipelines).

We continue using Lightning’s ModelCheckpoint and LearningRateMonitor callbacks to automatically track the best model and log learning-rate schedules during training.

Configuration Extensions

Lesson 2 adds new YAML files to support advanced features, but again, no source code changes are required. Each Hydra run is automatically logged to outputs/YYYY-MM-DD/HH-MM-SS/, keeping experiments isolated and reproducible.

Your directory now includes:

configs/
    trainer/
        single_gpu.yaml
        ddp.yaml
        fsdp.yaml

These configurations enable:

• single_gpu.yaml: local training
• ddp.yaml: distributed GPU training
• fsdp.yaml: model sharding for large models
• future extensions (AMP, gradient accumulation, etc.)

Because of Hydra’s override system, switching between them is as simple as:

python src/train.py trainer=ddp
python src/train.py trainer=fsdp
python src/train.py trainer=single_gpu

No code changes. No rewriting loops. No new training scripts.

This is exactly why Lightning and Hydra are so powerful: the same codebase can serve research, training at scale, and export pipelines without branching or duplication.

Enabling Mixed Precision Training with PyTorch Lightning AMP

Mixed precision training is one of the easiest “free wins” you can get when training Transformer models. Instead of doing every operation in full 32-bit floating point (FP32), we let the GPU run most of the math in 16-bit (FP16 or BF16) while keeping a few critical values in FP32 for numerical stability. The result is faster training, lower memory usage, and almost identical accuracy, especially on modern GPUs and Apple Silicon.

PyTorch Lightning wraps all of this inside the Trainer so you do not have to touch autocast contexts or manual gradient scaling. In this lesson, we will enable mixed precision purely through Hydra configurations, and Lightning will handle the rest.

Why Mixed Precision Boosts Transformers

Transformer models are heavy on matrix multiplications, which GPUs are extremely good at accelerating in half precision. When you switch from FP32 to a mixed precision mode (e.g., 16-mixed or bf16-mixed):

• Many tensor operations run 1.5-2.5× faster.
• Activations and gradients consume roughly half the memory.
• You can often increase batch size without running out of video random-access memory (VRAM).

Lightning uses PyTorch’s Automatic Mixed Precision (AMP) under the hood, so you still get stable training through automatic loss scaling. From an MLOps point of view, this is a small configuration change that can dramatically reduce training time and GPU cost without changing your codebase.

Updating Hydra: precision: 16-mixed

You do not need to modify data_module.py or model_module.py. Mixed precision is controlled entirely through the trainer configuration.

Here is the single-GPU trainer configuration in configs/trainer/single_gpu.yaml:

# Single GPU Configuration with Mixed Precision
# For experimentation and development

strategy: auto
devices: 1
accelerator: auto

# Mixed precision for faster training
precision: 16-mixed

# Training configuration
max_epochs: 3

# Performance
deterministic: false
benchmark: true

# Checkpointing
enable_checkpointing: true

# Progress
enable_progress_bar: true
log_every_n_steps: 10

Compared with an FP32 trainer, the required configuration change for mixed precision is:

precision: 16-mixed

Your DDP trainer in configs/trainer/ddp.yaml is already AMP-ready too:

# DDP (Distributed Data Parallel) Strategy
# Best for: Multi-GPU training on single node or multi-node setups
# Use when: Model fits in GPU memory, you want data parallelism
# Core distributed settings
strategy: ddp
devices: auto  # Use all available GPUs
accelerator: auto  # Auto-detect GPU/CPU

# Mixed precision for faster training
precision: 16-mixed  # FP16 automatic mixed precision

# Training configuration
max_epochs: 5  # More epochs for production
accumulate_grad_batches: 1  # Gradient accumulation (increase if OOM)

# Performance optimizations
deterministic: false  # Set to true for full reproducibility (slower)
benchmark: true  # Optimize CUDA kernels for input size

# Checkpointing
enable_checkpointing: true

# Progress tracking
enable_progress_bar: true
log_every_n_steps: 10

# DDP-specific optimizations
# ddp_find_unused_parameters: false  # Uncomment if you get DDP warnings

For FSDP, you are using BF16 mixed precision by default (better stability on very large models):

# FSDP (Fully Sharded Data Parallel) Strategy
# Best for: Very large models that don't fit in single GPU memory
# Use when: Model parameters exceed GPU memory, need memory efficiency

# Core distributed settings
strategy: fsdp
devices: auto  # Use all available GPUs
accelerator: auto

# Mixed precision - BF16 recommended for FSDP
precision: bf16-mixed  # Better stability than FP16 for large models

# Training configuration
max_epochs: 5
accumulate_grad_batches: 1

# Performance settings
deterministic: false
benchmark: true

# Checkpointing
enable_checkpointing: true

# Progress tracking
enable_progress_bar: true
log_every_n_steps: 10
# FSDP-specific settings
# Note: FSDP automatically shards model parameters, gradients, and optimizer states
# This allows training models that wouldn't fit on a single GPU

All of these configurations are passed to the Trainer through:

trainer = pl.Trainer(
    **cfg.trainer,
    callbacks=callbacks,
    logger=logger,
)

So switching between FP32, FP16, and BF16 is just a matter of changing precision in YAML (no changes to the training loop, no new context managers, and no extra boilerplate).

When to Use FSDP (Fully Sharded Data Parallel)

While DDP works well when the entire model fits on one GPU, FSDP is designed for cases where it doesn’t.

Instead of replicating the whole model on each GPU, FSDP shards parameters, gradients, and optimizer states across GPUs, allowing you to train models many times larger than a single GPU’s memory.

Lightning makes this just as simple as DDP:

python src/train.py trainer=fsdp

FSDP uses BF16 mixed precision by default (configured in configs/trainer/fsdp.yaml), which provides better stability for large-scale Transformer models.

Use FSDP if:

• you are hitting OOM even with small batch sizes
• you are experimenting with larger Transformer backbones
• you want memory-efficient training across multiple GPUs

Running AMP on NVIDIA GPUs or Apple Silicon

Once the configurations are in place, running AMP simply requires choosing the right trainer preset from the CLI.

Single GPU with Mixed Precision (NVIDIA GPU or Apple Silicon):

# From the lesson2 directory
python src/train.py trainer=single_gpu

Because trainer=single_gpu already sets:

accelerator: auto
devices: 1
precision: 16-mixed

Lightning will:

• Use CUDA if an NVIDIA GPU is available,
• Use MPS on a Mac with Apple Silicon,
• Fall back to the CPU otherwise (mixed precision provides little benefit there).

If you want to be explicit, you can override precision on the command line:

python src/train.py trainer=single_gpu trainer.precision=32        # FP32 baseline
python src/train.py trainer=single_gpu trainer.precision=16-mixed # FP16 AMP

The same pattern applies to DDP:

# Use all available GPUs with AMP
python src/train.py trainer=ddp

# Or force 2 GPUs with mixed precision
python src/train.py trainer=ddp trainer.devices=2 trainer.precision=16-mixed

For very large models using FSDP:

python src/train.py trainer=fsdp trainer.precision=bf16-mixed

In every case, you are only changing Hydra configuration values, while the underlying Lightning code path remains exactly the same.

What Speedups Should You Expect?

The exact speedup depends on your GPU, batch size, and model size, but for Transformer-style models like DistilBERT, you may see:

• 1.5-2× faster iteration times
• ~50% lower activation memory usage
• Similar validation accuracy to FP32

If you want to quantify performance in your own environment, you can run a simple timing experiment:

# FP32 (baseline)
time python src/train.py trainer=single_gpu trainer.precision=32 trainer.max_epochs=1

# FP16 mixed precision
time python src/train.py trainer=single_gpu trainer.precision=16-mixed trainer.max_epochs=1

You can compare the wall-clock time and VRAM usage between the 2 runs. In real MLOps pipelines, this can translate into lower training costs and the ability to run larger experiments on the same hardware.

Distributed Training with PyTorch Lightning DDP for Multi-GPU Scaling

Scaling your model across multiple GPUs is one of the fastest ways to reduce training time for Transformer architectures. PyTorch Lightning makes this dramatically easier by handling the boilerplate (process spawning, gradient synchronization, device management, and checkpoint coordination). In Lesson 2, we enable Distributed Data Parallel (DDP) using a Hydra configuration file and a single command-line override.

Let us walk through how this works and how you can run multi-GPU training without changing a single line of Python code.

What Distributed Data Parallel (DDP) Is

Distributed Data Parallel (DDP) is PyTorch’s recommended approach for training models across multiple GPUs. Each GPU receives:

• a full copy of the model
• a shard of the dataset
• parallel forward and backward passes

After each backward pass, gradients are synchronized across all GPUs so that every model replica remains synchronized.

The benefit is simple:

More GPUs → Larger global batch size → Faster training

Lightning abstracts the entire DDP workflow, so you do not need to manually write any multiprocessing logic, barrier synchronization, or device placement. All of this is handled by Trainer(strategy="ddp").

Hydra and Lightning Configuration (strategy: ddp)

DDP is enabled entirely through configuration, not code.

Here is the Hydra configuration file you include in Lesson 2:

configs/trainer/ddp.yaml

# Distributed training with DDP
strategy: ddp
accelerator: auto
devices: auto
precision: 16-mixed
max_epochs: 3

# Optional: helps stabilize multi-GPU runs
num_nodes: 1

This file replaces the default trainer settings when you select it from the CLI.

Lightning reads this configuration and automatically enables:

• automatic GPU detection
• multi-process spawning
• distributed samplers for the DataModule
• synchronized batch normalization
• gradient synchronization across devices
• safe checkpointing on rank 0

You do not need to modify train.py, DataModule, or LightningModule.

Running Multi-GPU Training

Use the following command to launch DDP:

$ python src/train.py trainer=ddp

To explicitly choose the number of devices:

$ python src/train.py trainer=ddp trainer.devices=2

To train on all visible GPUs:

$ python src/train.py trainer=ddp trainer.devices=auto

You can also combine overrides:

$ python src/train.py trainer=ddp trainer.precision=bf16-mixed data.batch_size=16

Hydra composes the configurations, Lightning spawns the GPU workers, and DDP executes the training loop.

No additional code is required.

Gradient Accumulation for Large Effective Batch Sizes

Modern Transformer models often benefit from larger batch sizes because they produce smoother gradients, more stable optimization, and sometimes higher accuracy in fewer iterations. However, large batches require more GPU memory, and even a mid-sized GPU may not be able to process them directly.

Gradient accumulation solves this by splitting a large batch across multiple smaller forward passes. Lightning collects gradients over several steps before performing one optimizer update, giving you the effect of large-batch training without increasing memory usage.

Why Gradient Accumulation Helps

Instead of running:

• Batch size = 64 (requires large GPU memory)

…you can simulate it as:

• batch_size = 8
• accumulate_grad_batches = 8

Lightning will:

• Run 8 forward and backward passes
• Accumulate gradients internally
• Call optimizer.step() only once

This produces the same gradient update you would get from a single batch of size 64 while using only the memory required for a batch of 8.

This is especially useful when:

• You hit CUDA OOM errors during DDP or FSDP training
• You want larger effective batch sizes for stability
• You are training with mixed precision, which can benefit from larger batches
• You are using limited GPU hardware, including Apple Silicon

Updating Hydra Configurations

Your Lesson 2 repository supports gradient accumulation through Hydra overrides.

The primary trainer configurations (ddp.yaml, fsdp.yaml, single_gpu.yaml) set:

accumulate_grad_batches: 1

This means accumulation is opt-in and controlled by command-line overrides.

You can also create a dedicated Hydra configuration:

configs/trainer/accum.yaml (optional)

accumulate_grad_batches: 4

However, this is not required because the command-line interface (CLI) override is often clearer and more flexible.

Using Gradient Accumulation via CLI

Because Hydra merges configurations top-down, you can override accumulation using the following options:

Single GPU

python src/train.py trainer=single_gpu trainer.accumulate_grad_batches=4

DDP (multi-GPU)

python src/train.py trainer=ddp trainer.accumulate_grad_batches=4

With Mixed Precision

python src/train.py trainer=ddp trainer.precision=16-mixed trainer.accumulate_grad_batches=8

With Large Batches

python src/train.py trainer=ddp data.batch_size=4 trainer.accumulate_grad_batches=8

In this case:

effective_batch_size = batch_size × accumulate_grad_batches × num_gpus

For example, with:

• batch_size = 4
• accumulate_grad_batches = 8
• num_gpus = 2

the effective batch size is:

4 × 8 × 2 = 64

This configuration normally requires ~16-20 GB of GPU memory but is now possible even on a laptop GPU.

Batch Size vs Memory Tradeoffs

Gradient accumulation directly affects:

• Lower Memory Consumption: Only the small per-step batch must fit in memory.
• Equivalent Gradient Updates: The optimizer receives the same gradient update as full-size training.
• Slightly Longer Training Time: Lightning performs more forward and backward passes before each update. However, this approach is still more efficient than dealing with out-of-memory (OOM) errors or reducing the sequence length or tokenizer settings.

Lightning Integration (Zero Code Changes)

Your Lesson 2 train.py requires zero code modification.

Lightning handles:

• gradient scaling
• accumulation logic
• optimizer stepping
• multi-GPU gradient synchronization
• mixed-precision scaling

You control the behavior through Hydra configurations.

Exporting PyTorch Lightning Transformer Models to ONNX and TorchScript

Training a model is only half the story because real-world ML systems need fast, portable, and framework-agnostic inference. In Lesson 2, you extend your training pipeline to automatically export your best checkpoint into two production-ready formats:

• ONNX: for cross-platform, high-performance inference (ONNX Runtime, TensorRT, OpenVINO)
• TorchScript: for PyTorch-native, C++ or mobile deployment

The key design goal is that none of this export logic lives inside the LightningModule. Instead, the export code is isolated in train.py and driven entirely by Hydra configuration. This keeps your model clean and your training loop reusable in both research and production settings.

Why Export?

Exported models provide several benefits:

Fast inference: ONNX Runtime routinely delivers 2-5× faster CPU inference than PyTorch eager mode.

Portability: A single ONNX file runs on:

• Linux, macOS, and Windows
• mobile devices
• GPU runtimes (e.g., TensorRT)
• serverless platforms (AWS Lambda with ONNX Runtime)

Reproducible Production Pipelines: TorchScript provides a stable, serialized version of your model for:

• C++ backends
• embedded devices
• custom inference servers
• TorchServe

DVC-ready artifact tracking: Exports drop cleanly into:

artifacts/models/
artifacts/metrics/

These artifacts can be versioned and tracked, just like code.

Enabling Export via Hydra

Exports are controlled by your root configuration:

configs/config.yaml

# Export configuration (LESSON 2 feature)
export:
  enabled: true       # Enable model export after training
  onnx: true         # Export to ONNX format
  torchscript: true  # Export to TorchScript format

You can override these settings at runtime:

Export Both Formats

python src/train.py trainer=ddp export.enabled=true

Export Only ONNX

python src/train.py export.enabled=true export.torchscript=false

Export Only TorchScript

python src/train.py export.enabled=true export.onnx=false

Lightning handles training, ModelCheckpoint saves the best checkpoint, and then your script reloads that checkpoint for export.

How the Export Pipeline Works

During export, the script automatically reloads the best checkpoint saved by ModelCheckpoint, ensuring the exported ONNX or TorchScript files always correspond to the best validation score.

Your train.py contains 2 production-grade export utilities:

ONNX Export

export_to_onnx(best_model, onnx_path, cfg.data.max_length)

TorchScript Export

export_to_torchscript(best_model, ts_path, cfg.data.max_length)

Both functions:

• load the best checkpoint
• switch the model to evaluation mode
• move it to the CPU
• construct a dummy input of shape (1, max_length)
• save the exported file under artifacts/models/

This separation ensures:

• no modification to model_module.py
• clean Transformer traceability
• reproducibility across runs

ONNX Export Details

The utility inside train.py uses:

torch.onnx.export(
    model,
    (dummy_input_ids, dummy_attention_mask),
    str(output_path),
    opset_version=14,
    input_names=["input_ids", "attention_mask"],
    output_names=["logits"],
    dynamic_axes={
        "input_ids": {0: "batch_size", 1: "sequence_length"},
        "attention_mask": {0: "batch_size", 1: "sequence_length"},
        "logits": {0: "batch_size"},
    },
)

Key features:

Dynamic axes: Allow variable batch sizes and sequence lengths at inference time.

Opset 14: Compatible with ONNX Runtime and TensorRT.

Framework-agnostic: You can deploy the same .onnx file using:

• ONNX Runtime
• NVIDIA TensorRT
• OpenVINO
• Triton Inference Server
• AWS Lambda (serverless inference)

ONNX Runtime Inference Example

Readers can test the exported model using:

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("artifacts/models/sentiment_classifier.onnx")

input_ids = np.random.randint(0, 1000, (1, 128), dtype=np.int64)
attention_mask = np.ones((1, 128), dtype=np.int64)

outputs = session.run(None, {
    "input_ids": input_ids,
    "attention_mask": attention_mask
})

print("Logits:", outputs[0])

This should produce logits similar to those from PyTorch inference.

TorchScript Export Details

The utility inside train.py uses:

traced_model = torch.jit.trace(
    model,
    (dummy_input_ids, dummy_attention_mask)
)
traced_model.save(str(output_path))

TorchScript provides:

PyTorch-Native Deployment

Works with:

• TorchServe
• custom C++ services
• mobile runtimes
• embedded systems

Stable, Production-Safe Serialization

Unlike Python pickles, TorchScript provides a stable serialized format for multi-process and multi-node systems.

Artifacts Stored in a DVC-Ready Structure

After export completes, your script creates:

artifacts/
├── models/
│   ├── sentiment_classifier.onnx
│   ├── sentiment_classifier.pt
└── metrics/
    └── metrics.json

Your script also generates artifacts/metrics/metrics.json, which contains the final validation loss and accuracy. Your continuous integration and continuous deployment (CI/CD) pipeline, Data Version Control (DVC), or model registry can consume this file.

This folder is well suited for DVC tracking:

dvc add artifacts/models/
dvc add artifacts/metrics/

End-to-End Export Command

A typical production run:

python src/train.py \
    trainer=ddp \
    trainer.precision=16-mixed \
    export.enabled=true

produces:

Exported: sentiment_classifier.onnx
Exported: sentiment_classifier.pt
Saved metrics.json

Both formats are validated, versioned, and ready for deployment.

Zero Modifications to Model Code

Because the LightningModule remains unchanged, you maintain:

• readability
• testability
• modularity
• compatibility with future lessons (CI/CD, DVC, deployments)

The entire export logic is kept in one place.

What's next? We recommend PyImageSearch University.

Course information:
120+ total classes • 115+ hours of on-demand code walkthrough videos • Last updated: August 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 120+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 94+ Certificates of Completion
• ✓ 115+ hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this lesson, you transformed a simple sentiment-classification project into a scalable, production-ready training system using PyTorch Lightning and Hydra. You learned how to enable mixed precision to speed up training, use Distributed Data Parallel (DDP) to leverage multiple GPUs, and apply gradient accumulation to simulate large batch sizes without increasing memory usage. All of these upgrades were achieved without modifying your model or data code, demonstrating the benefits of Lightning’s abstraction and Hydra’s configuration-driven design.

You also implemented a clean and reusable export pipeline, allowing the same trained model to be saved as ONNX or TorchScript and used in lightweight, fast inference systems. With a small export utility built directly into the train.py script and a clear configuration structure, you now have a workflow that produces consistent, portable artifacts suitable for CI/CD, edge devices, or cloud deployment.

Taken together, these enhancements provide a robust foundation for real-world ML engineering. You have taken the same codebase from Lesson 1 and upgraded it with improved performance, scalability, and deployment-ready outputs. This prepares you for next steps such as deployment, optimization, monitoring, or integrating your models into full MLOps pipelines.

Citation Information

Singh, V. “Scaling, Optimizing, and Exporting Transformers with PyTorch Lightning,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/2xntj

@incollection{Singh_2026_scaling-optimizing-exporting-transformers-pytorch-lightning,
  author = {Vikram Singh},
  title = {{Scaling, Optimizing, and Exporting Transformers with PyTorch Lightning}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/2xntj},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Scaling, Optimizing, and Exporting Transformers with PyTorch Lightning appeared first on PyImageSearch.

Running Gemma 4 in the Browser with Transformers.js and WebGPU

In the first part of this series, we explored the architecture behind Gemma 4, including its multimodal design, Mixture-of-Experts routing, long-context capabilities, and the innovations that make it one of Google’s most capable open-weight model families.

In the second part, we put those capabilities into practice using Hugging Face Transformers, building multimodal applications that processed images, videos, audio, and structured outputs from a unified Python interface.

Next, we shifted our focus to deployment. In the previous tutorial, we explored multiple ways to run Gemma 4 locally using Ollama, llama.cpp, MLX, LM Studio, and Transformers.js. Along the way, we learned how to download models, perform local multimodal inference, and expose OpenAI-compatible APIs without relying on cloud-hosted services.

But there is one final step that takes local AI even further:

Can we run Gemma 4 entirely inside a web browser?

Until recently, running multimodal large language models required native applications, Python environments, or dedicated inference servers. Modern browser technologies, however, have changed that. With Transformers.js, ONNX Runtime, and hardware acceleration through WebGPU and WebAssembly, we can now execute state-of-the-art models directly inside modern web browsers without sending data to external servers.

Running models entirely in the browser offers several advantages. It improves user privacy by keeping data on the client device, reduces infrastructure costs by eliminating backend inference servers, enables offline AI experiences, and simplifies deployment by allowing applications to run anywhere a modern browser is available.

In this tutorial, we will build a fully client-side multimodal application using Transformers.js. We will learn how to load an ONNX version of Gemma 4 inside the browser, process image inputs, perform local inference using browser hardware acceleration, and build an interactive web application that runs entirely on the user’s machine.

By the end of this guide, you will understand how to deploy Gemma 4 as a browser-native AI application without Python, without a backend inference server, and without sacrificing the multimodal capabilities we’ve explored throughout this series.

This lesson is the 4th in a 5-part series on Google DeepMind’s Gemma 4:

1. Google DeepMind’s Gemma 4: MoE, Efficiency Tricks, and Benchmarks
2. Building Multimodal AI Applications with Gemma 4 and Transformers
3. Running Gemma 4 Locally: Ollama, llama.cpp, MLX, and More
4. Running Gemma 4 in the Browser with Transformers.js and WebGPU (this tutorial)
5. Lesson 5

To learn how to build fully client-side multimodal AI applications with Gemma 4, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Building a Browser-Based Gemma 4 AI Application

Unlike the previous tutorial, where we interacted with Gemma 4 through the terminal, we will build a browser-based application that allows users to provide multimodal inputs and run inference directly from a web page.

The application consists of 3 primary components:

• An input panel for entering image and audio URLs, uploading local files, and specifying the text prompt.
• A status panel that displays the current stage of the inference pipeline, download progress, and elapsed execution time.
• An output panel where Gemma 4 streams its response as it is generated.

Creating the HTML Document and Styling the Interface

We begin by creating a standard HTML5 document that serves as the foundation for our browser application.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<title>Gemma 4 · WebGPU</title>
<style>/* CSS styling */</style>
</head>
<body>

<header>
 <h1>Gemma 4 · E2B</h1>
 <span class="badge">WebGPU</span>
</header>

Here, <!DOCTYPE html> declares an HTML5 document, while the <html> element serves as the root of the page. The <head> section contains metadata such as the character encoding, viewport configuration, page title, and CSS styles that define the application’s appearance. The <body> section contains the visible content displayed in the browser, and the <header> provides a simple title and badge identifying the application.

The <style> section defines the application’s layout, typography, color palette, progress indicators, animations, and responsive behavior. Since the focus of this tutorial is browser-based AI inference rather than frontend development, we won’t discuss the styling in detail. Instead, we will concentrate on the JavaScript code that loads Gemma 4 and performs multimodal inference directly in the browser.

Creating the Application Layout

Next, we create the main layout of our browser application.

<div class="main">...</div>

The interface is divided into 3 primary sections that guide users through the entire inference workflow:

• Input Panel: Allows users to provide an image and audio either by entering a URL or uploading local files. It also includes a text area for entering the prompt and a button for loading the model and starting inference.
• Status Panel: Displays the progress of the inference pipeline, including processor initialization, model loading, input preprocessing, token generation, and download progress.
• Output Panel: Displays the response generated by Gemma 4 as it is streamed to the browser in real time.

This layout separates user inputs, model status, and generated outputs into dedicated sections, making the application easier to navigate while providing clear feedback throughout the inference process.

With the user interface in place, we can now implement the JavaScript code responsible for loading Gemma 4 and running multimodal inference directly inside the browser.

Loading Transformers.js for Browser-Based Gemma 4 Inference

With the user interface in place, we can begin implementing the browser-based inference pipeline. We start by importing the components required from Transformers.js and specifying the Gemma 4 model that will be loaded throughout the application.

<script type="module">
import {
 AutoProcessor,
 Gemma4ForConditionalGeneration,
 TextStreamer,
 load_image,
} from "https://cdn.jsdelivr.net/npm/@huggingface/transformers@4/dist/transformers.min.js";

const MODEL_ID = "onnx-community/gemma-4-E2B-it-ONNX";

Unlike the previous tutorial, where we installed Transformers.js locally using npm, the browser version imports the library directly from the jsDelivr CDN. This allows the application to download the required JavaScript modules automatically when the page loads, eliminating the need for a local Node.js environment or build process.

The imported classes serve the same purpose as before:

• AutoProcessor: loads the preprocessing pipeline for Gemma 4, preparing text, images, and audio before they are passed to the model.
• Gemma4ForConditionalGeneration: loads the ONNX version of Gemma 4 and provides the API used for multimodal generation.
• TextStreamer: streams generated tokens to the browser as they are produced, enabling real-time output instead of waiting for the entire response.
• load_image: downloads and preprocesses images from either URLs or user-uploaded files.

Finally, we define the MODEL_ID, which points to the Gemma 4 E2B Instruct ONNX model hosted on the Hugging Face Hub. This is the same model used in the previous tutorial, ensuring consistent behavior across both the Node.js and browser implementations.

Checking for WebGPU Support

Before loading Gemma 4, we first verify that the user’s browser supports WebGPU. Unlike the Node.js implementation, which executes the model on the CPU, browser-based inference relies on WebGPU to accelerate model execution using the client’s GPU.

// ── WebGPU check ──
if (!navigator.gpu) {
 document.getElementById("error-msg").textContent = "WebGPU is not supported in this browser. Please use Chrome 113+ or Edge 113+.";
 document.getElementById("error-msg").classList.add("visible");
 document.getElementById("run-btn").disabled = true;
 document.getElementById("run-btn").textContent = "WebGPU not available";
}

The browser exposes WebGPU support through the navigator.gpu property. If this property is unavailable, the application assumes that the browser or hardware does not support GPU-accelerated inference.

Rather than allowing the application to continue and fail later when loading the model, we perform this check upfront. If WebGPU is unavailable, the application:

• displays an informative error message
• disables the Run Inference button
• prevents the user from attempting to load Gemma 4 on an unsupported browser

This early validation provides a better user experience by clearly communicating the requirements for running the application.

Note: At the time of writing, WebGPU is supported by modern Chromium-based browsers such as Google Chrome and Microsoft Edge. If your browser does not support WebGPU, consider updating to the latest version or enabling the appropriate experimental features.

Referencing the User Interface

Next, we create references to the HTML elements that we will interact with throughout the application.

// ── UI refs ──
const runBtn       = document.getElementById("run-btn");
const outputEl     = document.getElementById("output-text");
const errorEl      = document.getElementById("error-msg");
const progressWrap = document.getElementById("progress-wrap");
const progressFill = document.getElementById("progress-fill");
const progressPct  = document.getElementById("progress-pct");
const progressLbl  = document.getElementById("progress-label-text");
const elapsedEl    = document.getElementById("elapsed-text");
const imgPreview   = document.getElementById("img-preview");
const imgUrlInput  = document.getElementById("image-url");

These variables store references to the application’s user interface elements, including the Run Inference button, output panel, progress bar, status indicators, image preview, and input fields.

By retrieving these elements once at the beginning of the script, we can efficiently update the interface during model loading and inference without repeatedly querying the Document Object Model (DOM). This keeps the code cleaner, improves readability, and avoids unnecessary DOM lookups throughout the application.

Creating Helper Functions

Next, we define a few helper functions that simplify updating the application’s user interface during model loading and inference.

// ── Step helpers ──
function setStep(n, state) { // state: 'active' | 'done' | ''
 const el = document.getElementById(`step-${n}`);
 el.className = "step" + (state ? ` ${state}` : "");
}
function setProgress(pct, label) {
 progressWrap.classList.add("visible");
 progressFill.style.width = `${pct}%`;
 progressPct.textContent = `${Math.round(pct)}%`;
 if (label) progressLbl.textContent = label;
}
function showError(msg) {
 errorEl.textContent = msg;
 errorEl.classList.add("visible");
}
function clearError() { errorEl.classList.remove("visible"); }

Each helper is responsible for a specific aspect of the interface:

• setStep(): updates the status indicator for each stage of the inference pipeline, allowing users to see whether a step is currently running or has completed
• setProgress(): updates the model download progress bar and percentage value while the ONNX model is being downloaded and initialized
• showError(): displays error messages whenever model loading or inference fails
• clearError(): removes any previously displayed error messages before a new operation begins

Although these functions are not directly involved in running Gemma 4, they provide real-time feedback throughout the application’s execution, making it easier for users to monitor model loading and inference progress.

Previewing Images from a URL

To provide immediate visual feedback, we update the image preview whenever the user enters a new image URL.

// ── Image preview from URL ──
imgUrlInput.addEventListener("input", () => {
 const url = imgUrlInput.value.trim();
 if (url) { imgPreview.src = url; imgPreview.classList.add("visible"); }
 else imgPreview.classList.remove("visible");
});
// show default on load
imgPreview.src = imgUrlInput.value;
imgPreview.classList.add("visible");

This event listener monitors changes to the Image URL input field. Whenever the user enters a valid URL, the application updates the preview image so that the selected input can be verified before running inference. If the input is cleared, the preview is automatically hidden.

When the page first loads, the script also displays the default sample image specified in the input field, allowing users to run the application immediately without providing their own image.

Processing Local Image and Audio Inputs for Gemma 4 Multimodal AI

In addition to accepting image and audio URLs, our application also allows users to upload local files directly from their computer. This provides greater flexibility by supporting both online resources and locally stored media.

// ── File upload handling ──
let uploadedImageData = null; // base64 data URL
let uploadedAudioBuffer = null; // Float32Array

document.getElementById("img-file").addEventListener("change", (e) => {
 const file = e.target.files[0];
 if (!file) return;
 const reader = new FileReader();
 reader.onload = (ev) => {
   uploadedImageData = ev.target.result;
   imgPreview.src = uploadedImageData;
   imgPreview.classList.add("visible");
   imgUrlInput.value = "";
 };
 reader.readAsDataURL(file);
});

When an image is uploaded, the browser reads the selected file using the FileReader API and converts it into a Base64-encoded data URL. The resulting image is stored in uploadedImageData, displayed in the preview panel, and later passed to Transformers.js for inference.

document.getElementById("audio-file").addEventListener("change", async (e) => {
 const file = e.target.files[0];
 if (!file) return;
 const arrayBuffer = await file.arrayBuffer();
 const audioCtx = new AudioContext({ sampleRate: 16000 });
 const decoded = await audioCtx.decodeAudioData(arrayBuffer);
 uploadedAudioBuffer = decoded.getChannelData(0);
 document.getElementById("audio-url").value = "";
});

For audio files, the preprocessing pipeline is slightly different.

Instead of using FileReader, the browser loads the audio into an ArrayBuffer and decodes it using the Web Audio API (AudioContext). The decoded waveform is then extracted as a Float32Array using getChannelData(0) and stored in uploadedAudioBuffer.

This preprocessing step converts the uploaded audio into the numerical waveform representation expected by Gemma 4. By handling image and audio uploads within the browser, users can perform multimodal inference on their own files without relying on external servers or additional preprocessing tools.

Processing Audio Inputs in the Browser with Web Audio API

Earlier, we implemented a helper function for loading audio in a Node.js environment. In the browser, however, we can leverage the Web Audio API, which provides native support for decoding and processing audio files.

// ── Load audio via AudioContext (browser-native) ──
async function loadAudioBrowser(url) {
 const res = await fetch(url);
 const arrayBuffer = await res.arrayBuffer();
 const audioCtx = new AudioContext({ sampleRate: 16000 });
 const decoded = await audioCtx.decodeAudioData(arrayBuffer);
 return decoded.getChannelData(0); // Float32Array @ 16kHz
}

This helper first downloads the audio file using the Fetch API and stores it as an ArrayBuffer. The browser’s AudioContext then decodes the audio and resamples it to 16 kHz, matching the sampling rate expected by Gemma 4. Finally, getChannelData(0) extracts the waveform as a Float32Array, which is passed directly to the processor during inference.

Using the browser’s native audio processing capabilities eliminates the need for external libraries while providing the model with audio in the format required for multimodal inference.

Loading the Processor and Gemma 4 Model

With the user interface and helper functions in place, we can now load the processor and Gemma 4 model. Since downloading and initializing a large multimodal model can take some time, the application also displays a progress bar and status updates throughout the loading process.

// State
let processor, model;
let modelLoaded = false;

async function loadModel() {
    ...
}

We begin by defining three variables that manage the model’s state. The processor and model variables store the initialized processor and Gemma 4 model, while the modelLoaded flag tracks whether the model has already been loaded. This prevents the application from downloading and initializing the model multiple times during a single browser session.

The loadModel() function is responsible for downloading, initializing, and preparing Gemma 4 for inference.

Initializing the Gemma 4 Multimodal Processor

The first step is loading the processor.

setStep(1, "active");

processor = await AutoProcessor.from_pretrained(MODEL_ID);

setStep(1, "done");

As in the previous tutorial, AutoProcessor.from_pretrained() downloads the preprocessing pipeline associated with Gemma 4. The processor prepares text, images, and audio before they are passed to the model, ensuring that all three modalities are converted into the format expected during inference.

The status panel is updated before and after loading so users can track the application’s progress.

Loading the Gemma 4 ONNX Model for WebGPU Inference

Once the processor is ready, we initialize the Gemma 4 model.

model = await Gemma4ForConditionalGeneration.from_pretrained(
    MODEL_ID,
    {
        dtype: "q4f16",
        device: "webgpu",
        progress_callback: (info) => {
            ...
        },
    }
);

Here, Gemma4ForConditionalGeneration.from_pretrained() downloads the ONNX model and initializes it for browser-based inference.

Several important parameters are specified:

• dtype: "q4f16": loads a 4-bit floating-point quantized version of Gemma 4, reducing memory consumption while maintaining strong inference performance.
• device: "webgpu": instructs Transformers.js to execute the model using the browser’s WebGPU backend. Instead of performing inference on the CPU, computations are offloaded to the client’s GPU, resulting in significantly faster execution on supported hardware.
• progress_callback: receives download progress updates as the model files are fetched from the Hugging Face Hub. We use these updates to drive the application’s progress bar, allowing users to monitor the download in real time.

Since Gemma 4 consists of multiple ONNX weight files, the first execution may take several minutes depending on the user’s internet connection. Once downloaded, however, the browser caches these files, allowing subsequent executions to start much more quickly.

Updating the Browser Interface During Gemma 4 Model Loading

While the model is loading, the application continuously updates the elapsed time and download progress, giving users immediate feedback throughout the initialization process.

Once the download completes successfully, the progress indicator is updated, the Run Inference button is enabled, and the modelLoaded flag is set to true, indicating that Gemma 4 is ready to process user inputs.

If any step fails (e.g., due to a network issue or an unsupported browser), the exception is caught, an informative error message is displayed, and the application is reset so that the user can safely retry loading the model.

Running Gemma 4 Multimodal Inference Directly in the Browser

With the processor and model loaded, we are ready to perform multimodal inference. The application waits for the user to click the Run Inference button before preparing the inputs, generating a response, and streaming the output back to the browser.

// Run inference
runBtn.addEventListener("click", async () => {
    ...
});

The event listener serves as the entry point for the inference pipeline. Before performing any computation, it first checks whether Gemma 4 has already been loaded. If the model has not been initialized, the application automatically downloads and loads it. Otherwise, it immediately begins processing the user’s inputs.

Building the Prompt

The first stage constructs the multimodal conversation that will be passed to Gemma 4.

const prompt_text = document.getElementById("prompt-input").value.trim();

const messages = [{
    role: "user",
    content: [
        { type: "image" },
        { type: "audio" },
        { type: "text", text: prompt_text },
    ],
}];

const prompt = processor.apply_chat_template(
    messages,
    {
        enable_thinking: false,
        add_generation_prompt: true,
    }
);

As in the previous tutorial, we represent the input as a chat conversation. The user message contains placeholders for an image and an audio clip, along with the text instruction entered through the interface.

The conversation is then passed to apply_chat_template(), which formats the prompt according to Gemma 4’s expected conversational template. Setting enable_thinking to false disables thinking mode, while add_generation_prompt appends the appropriate generation token, indicating where the model should begin producing its response.

Processing Image and Audio Inputs with Transformers.js

Next, the application prepares the multimodal inputs.

// Load image
...

// Load audio
...

const inputs = await processor(
    prompt,
    image,
    audio,
    {
        add_special_tokens: false,
    }
);

The application supports both URLs and locally uploaded files. If the user has uploaded an image or audio clip, those inputs are used directly. Otherwise, the application downloads the resources from the URLs provided in the input fields.

Once the image, audio, and prompt are available, the processor converts them into the tensor representations expected by Gemma 4. This preprocessing step combines all three modalities into a unified set of model inputs, allowing the model to reason over text, images, and audio simultaneously.

Generating the Response

With the inputs prepared, we can finally generate a response.

const outputs = await model.generate({
    ...inputs,
    max_new_tokens: 512,
    do_sample: false,
    streamer: new TextStreamer(
        processor.tokenizer,
        {
            skip_prompt: true,
            skip_special_tokens: true,
            callback_function: (text) => {
                ...
            },
        }
    ),
});

The generate() method performs autoregressive text generation using the processed multimodal inputs.

Several parameters control the generation process:

• max_new_tokens: 512: limits the maximum number of tokens that Gemma 4 can generate.
• do_sample: false: disables stochastic sampling, producing deterministic responses for identical inputs.
• TextStreamer: streams generated tokens to the browser as soon as they are produced instead of waiting for the entire response to complete.

Inside the streamer’s callback function, each newly generated token is appended to the output panel, allowing users to watch the response appear in real time. This creates a significantly more interactive experience, particularly when generating longer responses.

Once generation completes, the application removes the animated cursor, updates the elapsed execution time, reports the total number of generated tokens, and re-enables the Run Inference button for the next request.

If an error occurs at any point during preprocessing or generation, the exception is caught, an informative error message is displayed, and the application is reset so that inference can be attempted again.

Output

After opening the HTML file in a WebGPU-enabled browser, you will see an interface similar to the one shown in Figure 1. The left panel allows you to provide an image and audio either by entering their URLs or by uploading local files. You can also customize the prompt that will be sent to Gemma 4.

Click Load Model to download the ONNX version of Gemma 4 into the browser. During the first run, the application displays the download progress along with the status of each stage in the inference pipeline. Once the model has been loaded, the button changes to Run Inference.

Clicking Run Inference processes the image, audio, and prompt entirely within the browser using WebGPU acceleration. As Gemma 4 generates its response, the output is streamed token by token into the output panel, allowing you to see the answer appear in real time rather than waiting for the entire generation to finish.

The first execution may take a few minutes because the model weights must be downloaded and initialized. Subsequent runs are significantly faster since the model is already cached by the browser, requiring only the preprocessing and inference steps to complete.

What's next? We recommend PyImageSearch University.

Course information:
120+ total classes • 115+ hours of on-demand code walkthrough videos • Last updated: August 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 120+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 94+ Certificates of Completion
• ✓ 115+ hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

In this tutorial, we explored how to run Gemma 4 entirely inside a web browser using Transformers.js and WebGPU, without relying on Python, Node.js, or any external inference server. Starting from a simple HTML page, we built a complete multimodal application capable of loading an ONNX version of Gemma 4, accepting image and audio inputs, tracking model download progress, and streaming responses directly in the browser.

Along the way, we learned how to initialize the model with AutoProcessor and Gemma4ForConditionalGeneration, verify WebGPU support, preprocess multimodal inputs, construct prompts using the chat template, and perform real-time token streaming with TextStreamer. By leveraging ONNX Runtime Web and the browser’s GPU, we were able to execute the entire inference pipeline locally while keeping all data on the user’s device.

With this tutorial, we have now covered the complete Gemma 4 deployment landscape, from Python applications using Hugging Face Transformers to local inference with Ollama, llama.cpp, MLX, LM Studio, Transformers.js for Node.js, and finally Transformers.js running directly in the browser. Together, these tutorials demonstrate how the same Gemma 4 model can be deployed across a wide range of environments, from research notebooks and desktop applications to fully client-side web experiences.

Citation Information

Thakur, P. “Running Gemma 4 in the Browser with Transformers.js and WebGPU,” PyImageSearch, S. Huot, G. Kudriavtsev, and A. Sharma, eds., 2026, https://pyimg.co/gx0fr

@incollection{Thakur_2026_running-gemma-4-in-browser-transformers-js-webgpu,
  author = {Piyush Thakur},
  title = {{Running Gemma 4 in the Browser with Transformers.js and WebGPU}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Georgii Kudriavtsev and Aditya Sharma},
  year = {2026},
  url = {https://pyimg.co/gx0fr},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Running Gemma 4 in the Browser with Transformers.js and WebGPU appeared first on PyImageSearch.

Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models

In the previous lesson, we explored the power of Qwen3’s text-only models for building intelligent chatbots capable of deep reasoning. However, the real world is fundamentally multimodal: humans do not just communicate through text; we perceive and interact with the world through vision and other sensory inputs. The ability to understand and reason about visual information alongside text represents a crucial step toward more capable and human-like AI systems. This is where Qwen3-VL (Vision-Language) enters the picture, bringing state-of-the-art multimodal understanding to the open-source community.

Qwen3-VL represents a significant advancement in vision-language models, building upon the robust foundations of the Qwen3 architecture while adding sophisticated visual understanding capabilities. Like its text-only counterpart, Qwen3-VL features both Instruct and Thinking variants, enabling users to choose between fast, direct responses and deep, step-by-step reasoning for complex visual understanding tasks.

The model can process and understand images alongside text queries, making it invaluable for applications (e.g., visual question answering, image captioning, complex scene understanding, and multi-image reasoning). Whether you are analyzing medical images, interpreting charts and diagrams, or building assistive technologies for visually impaired users, Qwen3-VL provides the foundational capabilities needed for production-ready multimodal systems.

In this lesson, we extend our chatbot from Lesson 1 by integrating Qwen3-VL’s vision-language capabilities. We will learn how to handle multimodal inputs where users can upload images alongside their text queries, and the chatbot will respond with contextually appropriate answers that demonstrate understanding of both modalities. Through practical implementation using the Hugging Face Transformers library and Gradio, we will build an interactive web interface that seamlessly processes image-text pairs.

The Thinking mode will reveal the model’s visual reasoning process, showing how it analyzes images step-by-step before arriving at conclusions. By the end of this lesson, you will have a fully functional multimodal chatbot that can engage in sophisticated conversations about visual content, laying the groundwork for even more advanced capabilities we will add in subsequent lessons.

This lesson is the 2nd in a 4-part series on Building with Qwen3 Models:

1. Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models
2. Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models (this tutorial)
3. Lesson 3
4. Lesson 4

To learn about Qwen3 and build intelligent applications from scratch, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Qwen3-VL Vision-Language Model: Architecture, Training, and Capabilities

Qwen3-VL Architecture Overview: SigLIP2 Vision Encoder and Multimodal Transformer Design

Qwen3-VL (Figure 1) builds upon the Qwen3 language model architecture by integrating a powerful vision encoder capable of processing both static images and dynamic video content. The model employs a vision encoder initialized from SigLIP2-SO-400M, containing approximately 543 million parameters specifically dedicated to visual understanding.

This vision encoder is trained on a carefully curated mixture of image and video data, ensuring strong performance across both image understanding tasks and video comprehension scenarios. The integration of vision and language modalities is achieved through an elegant architectural design that preserves the reasoning capabilities of the base language model while adding rich visual understanding.

The Qwen3-VL series offers multiple model sizes to accommodate different computational budgets and application requirements. The lineup includes compact models (e.g., Qwen3-VL-2B) designed for edge deployment and resource-constrained environments, mid-size variants (e.g., Qwen3-VL-4B and Qwen3-VL-8B) that balance performance and efficiency, and larger models up to Qwen3-VL-14B that push the boundaries of multimodal understanding.

Each model variant maintains the same architectural principles but scales the number of layers, attention heads, and hidden dimensions to achieve different capacity levels. All variants support extended context lengths, with smaller models handling 32K tokens and larger models supporting up to 128K tokens, enabling them to process long documents with multiple images or extended video sequences.

A key innovation in Qwen3-VL is the Time-aligned Multimodal Rotary Position Embedding (TM-RoPE), which extends the traditional Multimodal RoPE (M-RoPE) by incorporating explicit temporal information. TM-RoPE factorizes the conventional rotary position embedding into three distinct dimensions: temporal, height, and width. This design enables the model to effectively capture both fine-grained local temporal variations and long-range dependencies across video frames.

For text inputs, the three components share identical position identifiers, making TM-RoPE equivalent to one-dimensional RoPE. For images, a constant temporal ID is assigned to all visual tokens while their spatial positions determine the height and width IDs. For video, each frame receives a monotonically increasing temporal ID adjusted based on actual timestamps, ensuring consistent temporal resolution of 80 milliseconds per ID. This sophisticated positional encoding scheme enables seamless integration of information across diverse modalities.

The vision encoder processes visual inputs by first converting them into a format suitable for the transformer architecture. For images, the encoder divides the input into patches and projects these patches into the model’s hidden dimension. For videos, frames are sampled at a dynamic frame rate to preserve video information while aligning with audio sampling rates in multimodal scenarios. The vision encoder outputs a sequence of visual tokens that are then concatenated with text tokens and processed by the unified transformer backbone. This design allows the model to attend to both visual and textual information jointly, enabling rich cross-modal reasoning. The model can identify objects, understand spatial relationships, read text within images through OCR capabilities, interpret charts and diagrams, and reason about sequences of events in videos.

Both Qwen3-VL-Instruct and Qwen3-VL-Thinking variants are available, mirroring the dual-mode capability of the text-only models. The Instruct variant provides fast, direct responses to visual questions, making it ideal for applications requiring low latency (e.g., real-time image analysis or interactive user interfaces).

The Thinking variant generates explicit reasoning traces that reveal the model’s step-by-step visual analysis process. When presented with a complex image, the Thinking model verbalizes its observations, identifies relevant details, makes connections between visual elements, and systematically works toward a conclusion. This transparency is invaluable for applications requiring explainable AI, educational tools where understanding the reasoning process is important, and debugging or improving system performance.

Training Pipeline: Multimodal Pretraining with Image-Text and Video-Text Data

The training of Qwen3-VL follows a 4-stage approach (Table 1) that builds visual understanding capabilities on top of the strong language foundation established during Qwen3’s pretraining. The training corpus includes diverse sources of image-text pairs and video-text sequences, covering a wide range of domains, styles, and complexity levels.

The data encompasses natural images from everyday scenarios, professional photography, scientific charts and diagrams, screenshots of user interfaces, medical imagery, satellite imagery, artwork and historical photographs, and video sequences capturing dynamic events and actions. This diversity ensures the model develops robust visual understanding that generalizes well to novel scenarios.

The pretraining stage for Qwen3-VL employs a combination of objectives designed to align visual and textual representations. The primary training objective uses a contrastive learning approach similar to CLIP, where the model learns to associate images with their corresponding textual descriptions by maximizing the similarity between matching image-text pairs while minimizing similarity between non-matching pairs.

Additionally, the model is trained on image captioning tasks where it must generate descriptive text given an image, and visual question answering tasks where it must answer questions about image content. These complementary objectives encourage the model to develop both discriminative abilities (understanding what an image depicts) and generative abilities (producing accurate descriptions and responses).

A critical aspect of Qwen3-VL’s training is the integration of visual and language data at an early stage. Rather than training vision and language components separately and then attempting to bridge them, Qwen3-VL employs joint training from the beginning. This approach enables the model to learn shared representations that naturally capture cross-modal relationships.

The training data is carefully balanced to include both unimodal tasks (pure text or pure image understanding) and cross-modal tasks (image-text reasoning, visual question answering). This balance prevents modality-specific performance degradation (a common issue where improving one modality comes at the cost of another) and ensures the model maintains strong capabilities across all input types.

The vision encoder undergoes specialized training to extract meaningful features from images and videos. For static images, the encoder learns to identify salient objects, understand spatial relationships, recognize text through OCR, interpret abstract concepts like emotions or art styles, and extract information from specialized content (e.g., charts, graphs, and diagrams).

For video understanding, the encoder is trained to track objects across frames, understand temporal relationships and event sequences, recognize actions and activities, and maintain consistent object identity throughout clips. This comprehensive training regimen produces visual representations that are rich, detailed, and optimized for downstream reasoning tasks.

Post-training for Qwen3-VL follows a similar philosophy to the text-only models, with instruction tuning and reinforcement learning adapted for multimodal scenarios. The instruction tuning phase uses high-quality human-annotated image-text conversations where the model learns to follow user instructions that reference visual content.

These instructions span diverse task types (e.g., describing images in varying levels of detail, answering specific questions about image content, comparing multiple images, performing visual reasoning and inference, extracting structured information from images, and generating creative content inspired by images). The Thinking variant receives additional training on reasoning traces that explicitly verbalize the visual analysis process, teaching the model to describe what it observes, explain why certain observations are relevant, and show how visual evidence supports its conclusions.

Reinforcement learning fine-tuning (RLHF) further refines the model’s behavior based on human preferences. For multimodal tasks, this involves collecting feedback on the quality, accuracy, and relevance of responses to visual queries.

The model learns to prioritize responses that accurately describe visual content, provide helpful and relevant information, demonstrate appropriate levels of detail based on the query, acknowledge when visual information is ambiguous or unclear, and maintain consistency between visual observations and textual reasoning. For smaller Qwen3-VL models, knowledge distillation from larger variants transfers both visual understanding and reasoning capabilities, enabling compact models to achieve impressive performance despite their reduced parameter count.

Performance Benchmarks: Image, Video, and Multi-Image Reasoning Evaluation

Qwen3-VL demonstrates strong performance across a comprehensive suite of vision-language benchmarks, positioning it as a competitive open-source multimodal model. The model excels in traditional image understanding tasks (e.g., image classification, object detection, and visual question answering), while also showing strong capabilities in more challenging scenarios (e.g., multi-image reasoning, video understanding, and complex visual reasoning).

Across flagship benchmarks for evaluating vision-language models, Qwen3-VL consistently achieves results that rival or surpass much larger proprietary models, demonstrating the effectiveness of its architectural innovations and training methodology.

On general vision-language benchmarks, Qwen3-VL shows strong capabilities across multiple evaluation dimensions. For visual question answering tasks, the model demonstrates strong performance on datasets (e.g., ERQA and RealWorldQA) that require it to answer diverse questions about image content, ranging from simple object recognition to complex reasoning about relationships and attributes. On image captioning benchmarks (e.g., COCO and NoCaps), Qwen3-VL generates accurate, detailed, and fluent descriptions that capture both the main subjects and subtle details in images.

The model’s optical character recognition (OCR) capabilities are particularly strong (Figure 2), enabling it to accurately extract and understand text embedded within images across various fonts, layouts, and contexts.

Multi-image reasoning represents a particularly challenging category where models must integrate information across multiple images to answer questions or perform comparisons. Qwen3-VL excels at these tasks, demonstrating the ability to identify similarities and differences between images, track objects or concepts across a sequence of images, aggregate information from multiple sources, and reason about relationships between distinct visual inputs.

This capability is crucial for applications (e.g., analyzing time-series imagery, comparing before-and-after scenarios, or understanding narratives told through image sequences). The model’s performance on benchmarks specifically designed to test multi-image understanding consistently ranks among the top open-source alternatives.

For video understanding, Qwen3-VL processes sequences of frames to understand temporal dynamics and event structures. The model achieves strong results on video question answering benchmarks where it must comprehend actions, events, and temporal relationships depicted in video clips.

It demonstrates the ability to recognize and track objects throughout videos, understand cause-and-effect relationships between events, identify the sequence and timing of actions, and answer questions requiring temporal reasoning (“What happened before/after X?”). The integration of temporal information through TM-RoPE enables the model to maintain coherent understanding across extended video sequences, a capability that distinguishes it from models that treat videos as disconnected frames.

Specialized benchmarks reveal Qwen3-VL’s capabilities in domain-specific tasks. On mathematical and scientific diagram understanding (e.g., charts, graphs, and geometric figures), the model demonstrates the ability to extract numerical information from visualizations, understand mathematical notation and symbols, reason about spatial relationships in geometry problems, and interpret scientific diagrams and schematics.

For document understanding tasks (Figure 3), including complex layouts with mixed text and images, Qwen3-VL shows strong performance in extracting information from receipts, invoices, and forms, understanding document structure and hierarchy, reading tables and structured data, and processing multi-page documents with consistent context.

When comparing Instruct and Thinking variants, the Thinking model demonstrates particular advantages on tasks requiring complex reasoning or multi-step analysis. While the Instruct variant provides faster responses suitable for straightforward queries, the Thinking variant’s explicit reasoning traces lead to improved accuracy on challenging problems.

Users can observe the model systematically analyzing visual information, identifying relevant details, making intermediate inferences, and arriving at well-justified conclusions. This transparency not only improves performance on difficult tasks but also enables debugging, validation, and trust-building in deployment scenarios.

The Thinking variant shows the largest performance improvements over Instruct on tasks involving visual reasoning chains, ambiguous or complex images requiring careful analysis, questions that require synthesizing information from multiple parts of an image, and scenarios where explainability is valued alongside accuracy (Figure 4).

Implementation: Building a Multimodal Chatbot with Qwen3-VL and Gradio

Now, let us build on our foundation from Lesson 1 to create a multimodal chatbot that can understand and reason about images alongside text. We will integrate Qwen3-VL models and extend the Gradio interface to support image uploads.

Step 1: Installing Dependencies for Qwen3-VL and Multimodal Processing (PyTorch, Transformers, Gradio)

In addition to the packages from Lesson 1, we need a few more libraries for handling images and multimodal processing.

!pip install torch transformers accelerate gradio
!pip install flash-attn --no-build-isolation
!pip install qwen-vl-utils pillow
print("\n✅ Dependencies installed successfully!")

On Line 1, we install the same core dependencies as Lesson 1: PyTorch, Transformers, Accelerate, and Gradio. On Line 2, we include Flash Attention for optimized inference. On Line 3, we add 2 new packages: qwen-vl-utils provides utilities for processing vision inputs specific to Qwen3-VL models, including helper functions for formatting multimodal conversations, and pillow is Python’s standard image-processing library for loading, manipulating, and displaying images. These additional dependencies enable our chatbot to seamlessly handle visual inputs alongside text. On Line 4, we print a confirmation message.

Step 2: Importing Required Libraries

We extend our imports from Lesson 1 to include image processing capabilities.

import torch
from transformers import pipeline
import gradio as gr
from typing import List, Tuple, Optional
from PIL import Image
import requests
from io import BytesIO
from qwen_vl_utils import process_vision_info
from IPython.display import display, Markdown

print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"CUDA device: {torch.cuda.get_device_name(0)}")

On Lines 1-9, we import all necessary modules for our multimodal application. Beyond the basic imports from Lesson 1, we add Image from PIL for image manipulation, requests and BytesIO for downloading and handling image data from URLs, and process_vision_info from qwen_vl_utils for preprocessing vision inputs in the format expected by Qwen3-VL models. The IPython display imports help us visualize images and formatted text in Jupyter notebooks. On Lines 11-14, we perform the same environment checks as before to verify PyTorch installation and GPU availability.

Step 3: Initializing Qwen3-VL Instruct and Thinking Image-Text-to-Text Pipelines

Now we initialize the Qwen3-VL pipelines that can process both images and text.

print("=" * 60)
print("Initializing Qwen3-VL-2B-Instruct and Qwen3-VL-2B-Thinking Text Generation Pipelines")
print("=" * 60)

qwen3vl_instruct = pipeline("image-text-to-text", model="Qwen/Qwen3-VL-2B-Instruct")
qwen3vl_thinking = pipeline("image-text-to-text", model="Qwen/Qwen3-VL-2B-Thinking")

print("\n✅ Pipelines initialized successfully!")

On Lines 1-3, we print informative messages to track initialization progress. On Lines 5 and 6, we initialize 2 pipelines using the "image-text-to-text" task type instead of "text-generation" from Lesson 1. This task type indicates that the pipeline accepts both images and text as input and generates text as output.

We use the Qwen3-VL-2B variants, which offer excellent performance while remaining lightweight enough for efficient inference. The models automatically download from Hugging Face if not cached locally (approximately 4-5 GB each for the 2B parameter versions). These pipelines will handle all the complexity of encoding images, combining them with text, and generating appropriate responses.

Step 4: Testing Multimodal Conversations with Image Uploads and Text Prompts

Let us test the models with a conversation that includes images to understand their multimodal capabilities.

# Initialize conversation history
conversation = []

# First message with image
img1 = Image.open(BytesIO(requests.get("https://media.istockphoto.com/photos/dogs-playing-picture-id481634156", timeout=10).content))
user_msg_1 = "Describe the image briefly."
conversation.append({"role": "user", "content": [{"type": "text", "text": user_msg_1}, {"type": "image", "image": img1}]})
display(Markdown(f"**User**: {user_msg_1}"))
display(img1)

response_1 = qwen3vl_instruct(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
response_1 = response_1[0]['generated_text']
display(Markdown(f"\n**Assistant**: {response_1}\n"))

# Update history
conversation.append({"role": "assistant", "content": [{"type": "text", "text": response_1}]})

# Second message with different image (continuation)
img2 = Image.open(BytesIO(requests.get("https://th.bing.com/th/id/OIP.i7RVdGQWnsiTssgbUeCmsAHaLG", timeout=10).content))
user_msg_2 = "Differentiate between first and second image."
conversation.append({"role": "user", "content": [{"type": "text", "text": user_msg_2}, {"type": "image", "image": img2}]})
display(Markdown(f"**User**: {user_msg_2}"))
display(img2)

response_2 = qwen3vl_thinking(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
trace = response_2[0]['generated_text'].split('</think>')[0]
response_2 = response_2[0]['generated_text'].split('</think>')[1]
response_2 = (
    "<details><summary>Show trace</summary>\n\n"
    "```text\n" + (trace or "").strip() + "\n```\n"
    "</details>\n\n"
    + (response_2 or "")
)
display(Markdown(f"\nAssistant: {response_2}"))

On Line 2, we initialize an empty conversation list as before. On Lines 5-7, we download an image from a URL using the requests library, load it into a PIL Image object, and create the first user message. Note the new format on Line 7: instead of a simple string, the content is now a list containing both text and image dictionaries. This structure allows us to include multiple modalities in a single message. On Lines 8 and 9, we display both the text query and the image for visibility.

On Lines 11-13, we generate a response using the Instruct model with the same parameters as Lesson 1. The model processes both the text query and the image, generating a description that demonstrates visual understanding. On Line 16, we update the conversation history with the assistant’s response, formatting it as a text-type content entry.

On Lines 19-23, we introduce a second image and ask a comparative question that requires the model to recall information from the first image while analyzing the second. This tests the model’s ability to maintain visual context across conversation turns. On Lines 25-33, we switch to the Thinking model for the response. As in Lesson 1, we parse the thinking trace from the final answer and format it as collapsible HTML. The thinking trace now reveals not just textual reasoning but also how the model analyzes visual information: describing what it observes in each image, identifying key differences, and building toward a coherent comparative answer.

Figure 5 shows the output of the code snippet above, demonstrating the reasoning capabilities of both models.

Step 5: Building a Gradio Multimodal Chat Interface with Image Input Support

Now we will build a comprehensive Gradio interface that supports image uploads alongside text input.

import base64

def create_chatbot_interface(chatbot, thinking_chatbot):
    """
    Gradio interface for multimodal chatbot interaction
    """

    def pil_to_base64(img):
        buf = BytesIO()
        img.save(buf, format="PNG")
        return "data:image/png;base64," + base64.b64encode(buf.getvalue()).decode()

    def add_user_message(message: str, image, conversation, history):
        if not message.strip():
            return history, ""
        if history is None:
            history = []
        conversation.append({"role": "user", "content": [{"type": "text", "text": message}]})
        if image is not None:
            conversation.append({"role": "user", "content": [{"type": "image", "image": image}]})

        display_text = message or ""
        if image is not None:
            img_str = pil_to_base64(image)
            display_text += f"\n\n![uploaded image]({img_str})"
        history.append([display_text, None])

        return history, "", None   # show user message immediately

On Line 1, we import the base64 module for encoding images as strings. On Lines 3-6, we define our main interface function with a docstring explaining its purpose. On Lines 8-11, we define a helper function pil_to_base64 that converts PIL images to base64-encoded strings. This is necessary because we want to display uploaded images directly in the chat history. The function resizes images to 256×256 pixels for efficient display and encodes them as PNG data URLs.

On Lines 13-28, we define the add_user_message function, which now handles both text and optional image inputs. On Lines 14-17, we validate that the message is not empty and initialize history if needed. On Lines 18-20, we append the text content to the conversation and, if an image is provided, append it as a separate content entry with type "image".

This format matches what Qwen3-VL expects. On Lines 22-26, we create a display string for the chat history. If an image was uploaded, we convert it to a base64 data URL and embed it using markdown image syntax, allowing Gradio to render it directly in the chatbot UI. On Line 28, we return the updated history, clear the text input, and clear the image input, providing immediate visual feedback.

def chat_function(message: str, conversation, history, model_choice: str):
        try:
            if model_choice == "Thinking":
                response = thinking_chatbot(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
                trace = response[0]['generated_text'].split('</think>')[0]
                response = response[0]['generated_text'].split('</think>')[1]
                assistant_text = (
                    "<details><summary>Show thinking trace</summary>\n\n"
                    "```text\n" + (trace or "").strip() + "\n```\n"
                    "</details>\n\n"
                    + (response or "")
                )
            else:
                response = chatbot(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
                response = response[0]['generated_text']
                assistant_text = response or ""

            conversation.append({"role": "assistant", "content": [{"type": "text", "text": assistant_text}]})
            history[-1][1] = assistant_text
            return history, "", None
        except Exception as e:
            history[-1][1] = f"Exception: {e}"
            return history, "", None

On Lines 1-23, we define the core chat function that generates responses. The logic is nearly identical to Lesson 1, but now it handles multimodal conversation contexts. On Lines 3-12, if Thinking Mode is selected, we generate a response with reasoning traces and format them as collapsible details. On Lines 13-16, if Instruct Mode is selected, we generate a direct response. The models automatically process any images in the conversation context without us needing to explicitly pass them again, since the conversation list maintains the complete multimodal history. On Lines 18-23, we update the conversation with the assistant’s response and handle any exceptions gracefully.

    with gr.Blocks() as interface:
        gr.Markdown("## 🤖 Qwen3-VL-2B Multimodal Chatbot with Deep Thinking")

        with gr.Row():
            chatbot_ui = gr.Chatbot(scale=4, height=1000)

        with gr.Row():
            msg = gr.Textbox(placeholder="Type your message here...", scale=8)
            image_input = gr.Image(
                label="Upload Image (optional)",
                type="pil",
                scale=1
            )
            send_btn = gr.Button("📤 Send", scale=1)
            stop_btn = gr.Button("⏹️ Stop", scale=1)
            model_selector = gr.Dropdown(
                choices=["Instruct", "Thinking"],
                value="Instruct",
                label="Model selection",
                scale=2
            )

        history_state = gr.State([])
        conversation_state = gr.State([])

        # Step 1: Show user message immediately
        send_btn.click(
            add_user_message,
            inputs=[msg, image_input, conversation_state, history_state],
            outputs=[chatbot_ui, msg, image_input],
        )

        # Step 2: Then process with chatbot
        send_event = send_btn.click(
            chat_function,
            inputs=[msg, conversation_state, history_state, model_selector],
            outputs=[chatbot_ui, msg, image_input],
        )

        stop_btn.click(None, None, None, cancels=[send_event])

    return interface

On Lines 1 and 2, we create the Gradio interface with an updated title reflecting multimodal capabilities. On Lines 4 and 5, we create the chatbot display area. On Lines 7-21, we build the input row with several components: a text box for messages (Line 8), an image upload widget that accepts PIL images (Lines 9-13), send and stop buttons (Lines 14 and 15), and a model selector dropdown (Lines 16-21). The type="pil" parameter on the image input ensures uploaded images are automatically converted to PIL Image objects.

On Lines 23 and 24, we initialize state variables for tracking conversation history and display state. On Lines 27-31, we configure the send button’s first action to immediately display the user’s message and uploaded image. Note that image_input is now included in both inputs and outputs, allowing us to clear it after sending. On Lines 34-38, we configure the send button’s second action to process the message with the selected model. On Line 40, we wire up the stop button to cancel ongoing generation.

Step 6: Launching the Qwen3-VL Multimodal Chatbot Application

Finally, we create and launch our multimodal chatbot interface.

# Create the interface
interface = create_chatbot_interface(qwen3vl_instruct, qwen3vl_thinking)
print("✅ Gradio interface created!")

print("Launching Gradio interface...")
print("Access the chatbot at: http://localhost:7860")
print("Press the 'Stop' button in the output to stop the server\n")

interface.launch()

On Line 2, we instantiate the interface by passing both VL model pipelines to our creation function. On Lines 5-7, we print informative messages about accessing the application. On Line 9, we launch the Gradio interface on the default port. Users can now interact with a fully multimodal chatbot that understands and reasons about both text and images.

They can upload images individually or alongside text queries, ask questions about uploaded images, request comparisons between multiple images across the conversation, and switch between Instruct and Thinking Modes to see how the model analyzes visual information. The interface provides a seamless, intuitive way to explore Qwen3-VL’s powerful multimodal capabilities.

Multimodal Chatbot Results: Instruct Mode vs Thinking Mode Analysis

To illustrate how our chatbot adapts between Instruct Mode and Thinking Mode, we tested it on 2 different multimodal tasks. Below, we start with the concise instructive response, then show how the same system expands into deeper reasoning when Thinking Mode is enabled.

In Figure 6, we tried the chatbot in Instruct Mode, where it delivered a clear and balanced summary: sales across America, EMEA (Europe, the Middle East, and Africa), and Asia increased steadily over 4 years, with the strongest growth in Year 4. It also provided exact figures per region and year, but kept the explanation concise enough to avoid overwhelming detail.

This demonstrates how Instruct Mode excels at giving direct, useful insights from visual inputs, ideal for users who want quick answers.

In Figure 7, we uploaded a cartoon-style puzzle showing 2 forest scenes and asked the chatbot to “Spot the difference between both the images. Answer briefly.”

In Thinking Mode, the chatbot does not just state the differences; it walks through the reasoning: identifying the animals in each scene, comparing them, and concluding that the bottom image contains a zebra and an orange bird instead of the pink flamingo in the top image.

This shows how Thinking Mode provides step-by-step transparency, making the reasoning process visible and trustworthy.

What's next? We recommend PyImageSearch University.

Course information:
120+ total classes • 115+ hours of on-demand code walkthrough videos • Last updated: August 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 120+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 94+ Certificates of Completion
• ✓ 115+ hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

Throughout this lesson, we have explored Qwen3-VL and its remarkable multimodal capabilities that enable AI systems to understand and reason about visual information alongside text. We began by understanding the architectural innovations that make Qwen3-VL possible, including the SigLIP2-based vision encoder with 543 million parameters, the sophisticated TM-RoPE positional encoding that seamlessly integrates temporal, height, and width dimensions, and the joint training approach that prevents modality-specific performance degradation.

We learned how the model processes images by dividing them into patches, videos by sampling frames at dynamic rates, and maintains consistent understanding across extended sequences through absolute temporal encoding. The availability of both Instruct and Thinking variants provides flexibility between fast responses and transparent, step-by-step visual reasoning.

We then translated this theoretical knowledge into a practical implementation by extending our chatbot from Lesson 1 with multimodal capabilities. Through hands-on coding, we learned how to install and import additional dependencies for image processing, initialize vision-language pipelines using the “image-text-to-text” task type, structure multimodal conversations with content lists containing both text and image entries, and create a Gradio interface that supports image uploads alongside text input.

The implementation demonstrated key patterns for multimodal development: embedding images as base64 strings for display, maintaining visual context across conversation turns, parsing and displaying thinking traces for visual reasoning, and handling the complexity of mixed-modality inputs with clean, modular code. We saw firsthand how the Thinking model verbalizes its visual analysis, describing observations, identifying relevant details, and building toward well-reasoned conclusions.

This lesson represents a crucial step forward in our journey to build a comprehensive AI assistant. We have progressed from text-only interaction to sophisticated multimodal understanding, enabling applications such as visual question answering, image description and captioning, multi-image comparison and analysis, and video understanding through temporal reasoning.

In the next lesson (Lesson 3) of our series, we will add Qwen3-Guard to introduce safety moderation that protects users from harmful content in both text and image inputs. Following that (Lesson 4), we will integrate Qwen3-Embedding to enable retrieval-augmented generation, allowing our chatbot to search through knowledge bases and ground its responses in retrieved information. By combining these components (language understanding, visual reasoning, safety checks, and knowledge retrieval), we are building toward a production-ready multimodal AI assistant that demonstrates the full potential of the Qwen3 ecosystem.

Citation Information

Mangla, P. “Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/jf6oe

@incollection{Mangla_2026_building-multimodal-chatbot-qwen3-vl-instruct-thinking-models,
  author = {Puneet Mangla},
  title = {{Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/jf6oe},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Building a Multimodal Chatbot with Qwen3-VL Instruct and Thinking Models appeared first on PyImageSearch.

Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models

The landscape of artificial intelligence has witnessed remarkable progress in recent years, with large language models (LLMs) pushing the boundaries of what machines can understand and generate. Among the latest breakthroughs in this field is Qwen3, a comprehensive series of open-weight large language models that achieves state-of-the-art performance across diverse tasks and domains. Qwen3 represents a significant leap forward from its predecessor, Qwen2.5, introducing revolutionary features that fundamentally change how we interact with AI systems.

What sets Qwen3 apart is its unique dual-mode architecture that seamlessly integrates both thinking and non-thinking modes within a single model. This innovation eliminates the need to switch between different models for tasks requiring different levels of reasoning, such as moving from Qwen2.5 to QwQ for complex problem-solving.

The thinking mode enables deep, step-by-step reasoning through extended chain-of-thought (CoT) processes, making it ideal for challenging mathematical problems, complex coding tasks, and multi-step reasoning scenarios. Meanwhile, the non-thinking mode provides fast, direct responses for simpler queries where extensive deliberation is not necessary. This flexibility, combined with thinking budgets that allow fine-grained control over computational resources, makes Qwen3 a versatile tool for real-world applications.

In this lesson, we embark on an exciting journey to build an intelligent chatbot that leverages both Qwen3-Instruct and Qwen3-Thinking models. Through hands-on implementation, we will explore how these models can be integrated into a practical application using Gradio, a powerful framework for creating interactive web interfaces.

By the end of this tutorial, you will understand not only the theoretical underpinnings of Qwen3’s architecture and training methodology but also how to deploy these models in production-ready applications. This lesson serves as the foundation for our multi-part series on building a comprehensive, safe, and multimodal AI assistant using the complete Qwen3 model ecosystem.

This lesson is the 1st in the 4-part series on Building with Qwen3 Models:

1. Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models (this tutorial)
2. Lesson 2
3. Lesson 3
4. Lesson 4

To learn about Qwen3 and build intelligent applications from scratch, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Qwen3 Model Details

Qwen3 Model Architecture: Dense Models, MoE Variants, and Dual-Mode Reasoning

The Qwen3 series represents a comprehensive family of language models, offering both dense and Mixture-of-Experts (MoE) architectures to accommodate diverse computational requirements and use cases. The lineup includes 6 dense models ranging from the compact Qwen3-0.6B to the substantial Qwen3-32B, alongside two powerful MoE variants: Qwen3-30B-A3B and the flagship Qwen3-235B-A22B. The flagship model, with its 235 billion total parameters and 22 billion activated parameters per token, demonstrates the perfect balance between high performance and efficient inference through its sparse activation pattern.

The architecture of Qwen3 builds upon proven foundations while introducing critical enhancements for stability and performance. The dense models share core architectural elements with Qwen2.5, including Grouped Query Attention (GQA) for efficient attention computation, SwiGLU activation functions for better gradient flow, Rotary Positional Embeddings (RoPE) for superior position encoding, and RMSNorm with pre-normalization for training stability. However, Qwen3 introduces 2 significant improvements: the removal of QKV-bias that was present in Qwen2, and the introduction of QK-Norm to the attention mechanism, which ensures stable training especially for larger models.

The model variants span a wide range of sizes to meet different deployment scenarios (Table 1):

For the MoE models, Qwen3 adopts fine-grained expert segmentation with 128 total experts and 8 activated experts per token. Unlike Qwen2.5-MoE, the new design excludes shared experts and instead employs a global-batch load-balancing loss to encourage expert specialization (Table 2).

This architectural innovation has yielded substantial improvements in model performance across downstream tasks. The MoE models achieve performance comparable to much larger dense models while using only a fraction of the activated parameters, making them highly cost-effective for training and inference.

All Qwen3 models utilize Qwen’s tokenizer, which implements byte-level byte-pair encoding (BBPE) with a vocabulary size of 151,669 tokens. This comprehensive vocabulary enables efficient encoding across the 119 languages and dialects supported by Qwen3, from high-resource languages like English and Chinese to low-resource languages and specialized domains.

The combination of architectural innovations, diverse model sizes, and extensive multilingual support makes Qwen3 one of the most versatile and powerful open-source language model families available today.

Qwen3 Training Data and Post-Training Pipeline for Reasoning and Instruction Following

The training of Qwen3 represents a massive undertaking in data curation, processing, and model optimization. The pre-training dataset consists of approximately 36 trillion tokens (double the scale of Qwen2.5) and covers an unprecedented 119 languages and dialects compared to just 29 in the previous version. This expansion includes high-quality content spanning coding, STEM (Science, Technology, Engineering, and Mathematics), reasoning tasks, books, multilingual texts, and synthetic data, ensuring comprehensive knowledge coverage across domains.

To expand the pre-training corpus, the Qwen team employed innovative data augmentation strategies. First, they leveraged the Qwen2.5-VL model to perform text recognition on large volumes of PDF-like documents, extracting trillions of tokens from previously inaccessible sources. The recognized text was then refined using the Qwen2.5 model to improve quality.

Additionally, they employed specialized models (Qwen2.5-Math for mathematical content and Qwen2.5-Coder for code generation) to synthesize trillions of text tokens in various formats, including textbooks, question-answering pairs, instructions, and code snippets across dozens of domains. This synthetic data generation approach allowed for precise control over data quality and diversity.

A key innovation in Qwen3’s training is the development of a multilingual data annotation system that enhanced both quality and diversity. This system annotated over 30 trillion tokens across multiple dimensions such as educational value, fields, domains, and safety.

Unlike previous studies that optimized data mixture at the source or domain level, Qwen3’s approach optimized the data mixture at the instance level through extensive ablation experiments on small proxy models with fine-grained data labels. This granular approach to data curation significantly improved the effectiveness of the training process.

The pre-training process follows a carefully designed 3-stage strategy:

Stage 1. General Stage (S1): All Qwen3 models are trained on over 30 trillion tokens using a sequence length of 4,096 tokens. This stage establishes a strong foundation in language proficiency and general world knowledge across all 119 supported languages and dialects. The models learn fundamental patterns, linguistic structures, and broad knowledge that serves as the basis for more specialized capabilities.

Stage 2. Reasoning Stage (S2): To enhance reasoning abilities, the training corpus is optimized by increasing the proportion of STEM, coding, reasoning, and synthetic data. Models are further trained on approximately 5 trillion high-quality tokens at a sequence length of 4,096 tokens. The learning rate decay is accelerated during this stage to fine-tune the models’ reasoning capabilities, particularly for complex problem-solving tasks in mathematics, science, and programming.

Stage 3. Long Context Stage: In the final pre-training stage, high-quality long context corpora are used to extend the context length from 4,096 to 32,768 tokens. The long context corpus includes 75% of text between 16,384 and 32,768 tokens in length, and 25% of text between 4,096 and 16,384 tokens. The base frequency of RoPE is increased from 10,000 to 1,000,000 using the ABF technique, and YARN and Dual Chunk Attention (DCA) techniques are introduced to achieve a 4-fold increase in sequence length capacity during inference, enabling the models to handle documents up to 128K tokens.

Post-training (Figure 1) employs a sophisticated multi-stage approach to align the models with human preferences and downstream applications. The first 2 stages focus on developing strong reasoning abilities through long chain-of-thought (CoT) cold-start fine-tuning and reinforcement learning, primarily targeting mathematics and coding tasks.

In the final 2 stages, data with and without reasoning paths are combined into a unified dataset for further fine-tuning, enabling the model to handle both thinking and non-thinking modes effectively. General-domain reinforcement learning is then applied to improve performance across a wide range of downstream tasks.

For smaller models, strong-to-weak distillation leverages both off-policy and on-policy knowledge transfer from larger models, significantly outperforming direct reinforcement learning in both performance and training efficiency.

Qwen3 Benchmark Performance: Thinking Budgets, Reasoning, Coding, and STEM Tasks

Qwen3 demonstrates exceptional performance across a comprehensive suite of benchmarks, establishing new standards for open-source language models. The flagship model (Qwen3-235B-A22B) consistently outperforms alternatives across most benchmarks (Table 3).

When evaluated against DeepSeek-V3-Base, which has approximately three times more parameters (671B vs 235B) and nearly double the activated parameters (37B vs 22B), Qwen3-235B-A22B achieves superior results on 14 out of 15 evaluation benchmarks. This includes substantial advantages on general tasks like MMLU (87.81 vs 87.19), MMLU-Redux (87.40 vs 86.14), and BBH (88.87 vs 86.22), as well as specialized domains such as MATH (71.84 vs 62.62) and EvalPlus coding benchmarks (77.60 vs 63.75).

The performance advantages extend across the entire Qwen3 family. The MoE models demonstrate exceptional efficiency, achieving similar performance to dense models with only 1/5 of the activated parameters. For instance, experimental results show that Qwen3-30B-A3B (with only 3B activated parameters) performs comparably to much larger dense models, while Qwen3-235B-A22B outperforms the Qwen2.5 MoE base models with less than 1/2 of their activated parameters and fewer total parameters.

The dense Qwen3 models also show impressive scaling efficiency. Qwen3-1.7B/4B/8B/14B/32B-Base achieve performance comparable to Qwen2.5-3B/7B/14B/32B/72B-Base, respectively, effectively matching models that are 1.5-2x larger in parameter count (Table 4).

On STEM, coding, and reasoning benchmarks, Qwen3 dense base models frequently surpass their Qwen2.5 counterparts at higher parameter scales. For example, Qwen3-32B-Base scores 65.54 on MMLU-Pro compared to Qwen2.5-72B-Base’s 58.07, demonstrating substantial improvements in reasoning capabilities despite having less than half the parameters.

A particularly noteworthy finding is that increasing the thinking budget for thinking tokens leads to consistent improvements in model performance across various tasks (Figure 2). This demonstrates the effectiveness of the dual-mode architecture, where users can dynamically control the trade-off between response speed and reasoning depth based on their specific needs.

The combination of architectural innovations, extensive training data, and sophisticated optimization techniques positions Qwen3 as one of the most capable and efficient open-source language model families available, offering performance competitive with proprietary alternatives while maintaining the benefits of transparency and community-driven development.

Implementation: Building a Deep Thinking Chatbot with Qwen3 and Gradio

Now that we understand the theoretical foundations of Qwen3, let us dive into building a practical chatbot application that leverages both the Instruct and Thinking modes. We will create an interactive web interface using Gradio that allows users to seamlessly switch between fast responses and deep reasoning.

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

Step 1: Install Qwen3 Chatbot Dependencies with PyTorch, Transformers, and Gradio

First, we need to install the required Python packages for our chatbot. This includes PyTorch for model inference, Transformers for loading the Qwen3 models, and Gradio for building the web interface.

!pip install torch transformers accelerate gradio
!pip install flash-attn --no-build-isolation
print("\n✅ Dependencies installed successfully!")

On Lines 1-3, we install the core dependencies for our chatbot application. The torch package provides the deep learning framework for running model inference, while transformers from Hugging Face gives us easy access to pre-trained Qwen3 models.

The accelerate library optimizes model loading and inference across different hardware configurations, and gradio enables us to build an interactive web interface with minimal code. We also install flash-attn for optimized attention mechanisms that significantly speed up inference, particularly for long sequences. The --no-build-isolation flag ensures compatibility with the existing environment.

Step 2: Importing Required Libraries

Next, we import the necessary libraries and verify our environment setup.

import torch
from transformers import pipeline
import gradio as gr
from typing import List, Tuple, Optional
from IPython.display import display, HTML, Markdown

print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"CUDA device: {torch.cuda.get_device_name(0)}")

On Lines 1-5, we import the essential modules for our application. The torch import provides access to PyTorch functionality, while pipeline from transformers simplifies model loading and inference. Gradio (gr) will power our web interface, and the typing hints ensure code quality. The IPython display utilities help us render formatted output in Jupyter notebooks.

On Lines 7-10, we perform environment checks to verify that PyTorch is properly installed and to confirm whether GPU acceleration is available. If CUDA is available, we also print the GPU device name to ensure we are utilizing hardware acceleration for faster inference.

Step 3: Initialize Qwen3 Instruct and Qwen3 Thinking Text-Generation Pipelines

Now we create instances of both the Instruct and Thinking model pipelines. The pipeline abstraction from Hugging Face makes it incredibly easy to load and use these models.

print("=" * 60)
print("Initializing Qwen3-4B-Instruct and Qwen3-4B-Thinking Text Generation Pipelines")
print("=" * 60)

qwen3_instruct = pipeline("text-generation", model="Qwen/Qwen3-4B-Instruct-2507")
qwen3_thinking = pipeline("text-generation", model="Qwen/Qwen3-4B-Thinking-2507")

print("\n✅ Pipelines initialized successfully!")

On Lines 1-3, we print informative messages to track the initialization process. On Lines 5 and 6, we initialize 2 separate pipelines: one for the Instruct model and one for the Thinking model.

The pipeline function automatically handles model downloading (approximately 8GB for the 4B parameter model), tokenization, and inference setup. We use the Qwen3-4B variants, which offer an excellent balance between performance and resource requirements, making them suitable for a wide range of hardware configurations. The models are automatically downloaded from the Hugging Face model hub if they are not already cached locally.

Step 4: Test Qwen3 Instruct vs. Thinking Models in a Multi-Turn Conversation

Before building the full interface, let us test both models with a simple conversation to understand their behavior.

# Initialize conversation history
conversation = []

# First message
user_msg_1 = "What is machine learning? Explain shortly."
conversation.append({"role": "user", "content": user_msg_1})
display(Markdown(f"**User**: {user_msg_1}"))

response_1 = qwen3_instruct(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
response_1 = response_1[0]['generated_text']
display(Markdown(f"\n**Assistant**: {response_1}\n"))

# Update history
conversation.append({"role": "assistant", "content": response_1})

# Second message (continuation)
user_msg_2 = "Can you give me a simple short example?"
conversation.append({"role": "user", "content": user_msg_2})
display(Markdown(f"**User**: {user_msg_2}"))

response_2 = qwen3_thinking(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
trace = response_2[0]['generated_text'].split('</think>')[0]
response_2 = response_2[0]['generated_text'].split('</think>')[1]
response_2 = (
    "<details><summary>Show trace</summary>\n\n"
    "```text\n" + (trace or "").strip() + "\n```\n"
    "</details>\n\n"
    + (response_2 or "")
)
display(Markdown(f"\nAssistant: {response_2}"))

On Lines 1 and 2, we initialize an empty conversation list that will store the dialogue history in the format expected by the models. On Lines 4-7, we create the first user message asking about machine learning and append it to the conversation with the “user” role.

On Lines 9 and 10, we use the Instruct model to generate a response with do_sample=False for deterministic output and max_new_tokens=1024 to limit response length. The return_full_text=False parameter ensures we only get the newly generated text, not the entire prompt.

On Lines 13-18, we update the conversation history with the assistant’s response and then add a follow-up question. This demonstrates how the models maintain context across multiple turns. On Line 21, we switch to the Thinking model for the second response, which generates output with explicit reasoning traces.

On Lines 22-29, we parse the model’s output to separate the thinking process (enclosed in <think> tags) from the final answer. We format this as collapsible HTML details, allowing users to optionally view the model’s step-by-step reasoning process. This separation showcases the key difference between Instruct and Thinking modes: while Instruct provides direct answers, Thinking reveals its internal reasoning process.

Figure 3 shows the output of the above code snippet, demonstrating the reasoning capabilities of both models.

Step 5: Creating the Interactive Gradio Interface

Now we will build a comprehensive web interface that allows users to interact with both models through a clean, intuitive chat interface.

def create_chatbot_interface(chatbot, thinking_chatbot):
    """
    Gradio interface for chatbot interaction
    """

    def add_user_message(message: str, conversation, history):
        if not message.strip():
            return history, ""
        if history is None:
            history = []
        conversation.append({"role": "user", "content": message})
        history.append([message, None])
        return history, ""   # show user message immediately

    def chat_function(message: str, conversation, history, model_choice: str):
        try:
            if model_choice == "Thinking":
                response = thinking_chatbot(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
                trace = response[0]['generated_text'].split('</think>')[0]
                response = response[0]['generated_text'].split('</think>')[1]
                assistant_text = (
                    "<details><summary>Show thinking trace</summary>\n\n"
                    "```text\n" + (trace or "").strip() + "\n```\n"
                    "</details>\n\n"
                    + (response or "")
                )
            else:
                response = chatbot(conversation, do_sample=False, max_new_tokens=1024, return_full_text=False)
                response = response[0]['generated_text']
                assistant_text = response or ""

            conversation.append({"role": "assistant", "content": assistant_text})
            history[-1][1] = assistant_text
            return history, ""
        except Exception as e:
            history[-1][1] = f"Exception: {e}"
            return history, ""

On Lines 1-4, we define the main function that creates our chatbot interface, taking both the Instruct and Thinking model pipelines as parameters. On Lines 6-13, we define add_user_message, a helper function that handles user input. It validates that the message is not empty, initializes the history if needed, and appends the user’s message to both the conversation state and the display history. The function returns the updated history with the user’s message immediately visible, providing responsive feedback.

On Lines 15-37, we define the chat_function that processes messages and generates responses. The function first checks the model_choice parameter to determine which model to use. On Lines 17-26, if “Thinking” mode is selected, we generate a response using the thinking model and parse out the reasoning trace, formatting it as collapsible HTML for optional viewing.

On Lines 27-30, if “Instruct” mode is selected, we generate a straightforward response without thinking traces. On Lines 32-37, we update the conversation history with the assistant’s response and handle any exceptions gracefully by displaying error messages.

 with gr.Blocks() as interface:
        gr.Markdown("## 🤖 Qwen3-4B Chatbot with Deep Thinking")

        with gr.Row():
            chatbot_ui = gr.Chatbot(scale=4)

        with gr.Row():
            msg = gr.Textbox(placeholder="Type your message here...", scale=8)
            send_btn = gr.Button("📤 Send", scale=1)
            stop_btn = gr.Button("⏹️ Stop", scale=1)
            model_selector = gr.Dropdown(
                choices=["Instruct", "Thinking"],
                value="Instruct",
                label="Model selection",
                scale=2
            )

        history_state = gr.State([])
        conversation_state = gr.State([])

        # Step 1: Show user message immediately
        send_btn.click(
            add_user_message,
            inputs=[msg, conversation_state, history_state],
            outputs=[chatbot_ui, msg],
        )

        # Step 2: Then process with chatbot
        send_event = send_btn.click(
            chat_function,
            inputs=[msg, conversation_state, history_state, model_selector],
            outputs=[chatbot_ui, msg],
        )

        stop_btn.click(None, None, None, cancels=[send_event])

    return interface

On Lines 1 and 2, we use Gradio’s Blocks API to create a custom layout, starting with a markdown header. On Lines 4 and 5, we create the main chatbot display area with scale=4 to give it more screen space. On Lines 7-16, we build the input row containing a text box for user messages, send and stop buttons, and a dropdown menu for model selection. The scaling parameters control the relative widths of these elements for an optimal layout.

On Lines 18 and 19, we initialize 2 state variables: history_state for the chat display and conversation_state for the full conversation context needed by the models. On Lines 21-26, we configure the send button’s first action to immediately display the user’s message, providing instant feedback.

On Lines 28-33, we configure the send button’s second action to process the message with the selected model and generate a response. The send_event variable captures this event so we can cancel it if needed. On Line 35, we wire up the stop button to cancel ongoing generation, giving users control over long-running requests.

Step 6: Launch the Qwen3 Gradio Chatbot Application Locally

Finally, we create the interface instance and launch the web application.

# Create the interface
interface = create_chatbot_interface(qwen3_instruct, qwen3_thinking)
print("✅ Gradio interface created!")

print("Launching Gradio interface...")
print("Access the chatbot at: http://localhost:7860")
print("Press the 'Stop' button in the output to stop the server\n")

interface.launch()

On Lines 2 and 3, we instantiate our chatbot interface by calling create_chatbot_interface with both model pipelines and confirm successful creation. On Lines 5-7, we print informative messages about how to access the application.

On Line 9, we launch the Gradio interface, which starts a local web server on port 7860. The interface automatically opens in a new browser tab (or provides a link if running in certain environments). Users can now interact with both Qwen3-Instruct and Qwen3-Thinking models through an intuitive chat interface, seamlessly switching between fast responses and deep reasoning as needed.

Sample Qwen3 Chatbot Conversations

To demonstrate how our chatbot blends instructional clarity with deep reasoning, we have included three example interactions below. These showcase how the model responds in both normal and “Thinking Mode,” which users can toggle to reveal step-by-step logic and internal traceability.

In Figure 4, the chatbot responds to a basic science question: “Explain photosynthesis in simple terms.” What is notable here is the clarity and structure of the explanation. The model breaks the process into 4 intuitive steps: sunlight absorption, carbon dioxide intake, water uptake, and glucose production. It then concludes with a simplified chemical equation and adds a metaphor (“a natural kitchen”) to make the concept more relatable. This reflects the model’s instructional capability: concise, clear, and beginner-friendly.

Figure 5 shows the chatbot solving a math problem in Thinking Mode. Here, the chatbot first reveals its internal reasoning trace: a step-by-step breakdown of how it arrived at the answer. It recalls the formula, substitutes values, performs intermediate calculations, and even reflects on whether to include units. This traceability is key: it shows how the model thinks, not just what it knows. For users who want transparency, learning support, or deeper trust, this mode is invaluable.

Finally, in Figure 6, the chatbot provides the correct formula and numerical result, but also distinguishes between the exact answer (49π) and the approximate value (153.94). This demonstrates its ability to handle both symbolic and numeric reasoning.

What's next? We recommend PyImageSearch University.

Course information:
120+ total classes • 115+ hours of on-demand code walkthrough videos • Last updated: August 2026
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you'll find:

• ✓ 120+ courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 94+ Certificates of Completion
• ✓ 115+ hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Click here to join PyImageSearch University

Summary

Throughout this lesson, we have explored the groundbreaking Qwen3 model series and its unique dual-mode architecture that revolutionizes how we interact with large language models. We began by understanding the comprehensive model family, from the compact 0.6B parameter variant to the flagship 235B parameter MoE model, each designed to meet different computational requirements and use cases. The architectural innovations (including QK-Norm for training stability, fine-grained expert segmentation in MoE models, and support for 119 languages) demonstrate Qwen3’s position at the forefront of open-source language models.

The 3-stage training process, encompassing general knowledge acquisition, reasoning enhancement, and long-context capability development, produces models that excel across diverse benchmarks while maintaining efficiency through sparse activation patterns.

We then translated this theoretical knowledge into practical implementation by building a functional deep-thinking chatbot. Through hands-on coding, we learned how to set up the development environment, initialize both Instruct and Thinking model pipelines, and create an interactive web interface using Gradio. The implementation showcased key differences between the two modes: Instruct provides fast, direct responses ideal for straightforward queries, while Thinking reveals its step-by-step reasoning process, making it invaluable for complex problem-solving tasks.

We implemented features like conversation history management, dynamic model switching, and collapsible thinking traces, demonstrating best practices for building user-friendly AI applications. The modular code structure we developed provides a solid foundation that can be extended with additional features such as system prompts, temperature controls, or multi-user support.

This lesson serves as the cornerstone of our series on building a comprehensive, safe, and multimodal AI assistant. In the upcoming lessons, we will expand on this foundation by integrating Qwen3-VL for visual understanding (Lesson 2), adding Qwen3-Guard for safety moderation (Lesson 3), and incorporating Qwen3-Embedding for retrieval-augmented generation (Lesson 4). Each addition will transform our basic chatbot into an increasingly sophisticated system capable of understanding images, ensuring safe interactions, and retrieving relevant information from knowledge bases.

The skills and patterns established in this lesson (from pipeline initialization to interface design) will be reused and extended throughout the series. By the end of the complete series, we will have built a production-ready multimodal AI assistant that demonstrates the full potential of the Qwen3 ecosystem. This assistant will combine state-of-the-art language understanding, visual processing, safety guarantees, and knowledge retrieval in a single cohesive application that showcases the power of modern open-source AI.

Citation Information

Mangla, P. “Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/o0x9h

@incollection{Mangla_2026_building-intelligent-chatbot-qwen3-instruct-thinking-models,
  author = {Puneet Mangla},
  title = {{Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/o0x9h},
}

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Building an Intelligent Chatbot with Qwen3 Instruct and Thinking Models appeared first on PyImageSearch.