Home›Computer Vision Tutorial›Full Systems

Computer Vision · Lesson 88 of 111

Asking a Vision Model

And checking the answer.

⏱ 2 min read▶ Video at 2:18:13🎓 Free

❮ Previous Next ❯

Watch this lesson in the Computer Vision courseWatch on YouTube ↗Full Computer Vision course

Large models can now look at an image and answer questions, with no training from you. So why build anything? Why not send every photo to one? Let's put a real one to the test, and be honest about the answer.

Asking a Vision Model — From the video: Asking a Vision Model

Models can look at images
No training needed
Why build anything?

Real counts from a real model

We send three belt photos, with four, seven, and eleven apples, to a real model, and ask for a count, in a schema. We know the truth, so we can keep score. Run it. It got all three exactly right. That's impressive, with no training at all. It also says that it can see bruises. So for a quick prototype, this is often the fastest way to start. But three images is no proof. And think about the belt. Twelve apples a second means twelve network calls a second, each taking a second or more. It's far too slow, and too costly, for the line. And you can't tell it where the air jet should aim.

vision_llm.pypython

import cv2
from google import genai
from google.genai import types
from pydantic import BaseModel
from belt import belt

class Count(BaseModel):
    apples: int
    any_dark_bruise: bool

client = genai.Client()
right = 0
for seed, n in [(21, 4), (22, 7), (23, 11)]:
    img, _, _ = belt(n=n, seed=seed)
    photo = types.Part.from_bytes(data=cv2.imencode(".jpg", img)[1].tobytes(), mime_type="image/jpeg")
    r = client.models.generate_content(
        model="gemini-flash-latest", contents=[photo, "Count the apples on this belt."],
        config=types.GenerateContentConfig(temperature=0, response_mime_type="application/json",
                                           response_schema=Count))
    right += r.parsed.apples == n
    print(f"Truth {n:>2}, model says {r.parsed.apples:>2}, bruise seen: {r.parsed.any_dark_bruise}")
print(f"Exact counts: {right}/3")

Output

Truth  4, model says  4, bruise seen: True
Truth  7, model says  7, bruise seen: True
Truth 11, model says 11, bruise seen: True
Exact counts: 3/3

Also uses belt.py

import numpy as np
import cv2

def belt(n=7, seed=1, size=(480, 640), bruises=2, light=1.0):
    """Draw a simulated conveyor belt of apples. Returns the image and the true boxes."""
    rng = np.random.default_rng(seed)
    h, w = size
    img = np.full((h, w, 3), (70, 75, 80), np.uint8)
    img = cv2.add(img, rng.normal(0, 6, img.shape).clip(0, 30).astype(np.uint8))
    boxes, spots, taken = [], [], []
    while len(boxes) < n:
        r = int(rng.integers(32, 48))
        x, y = int(rng.integers(r + 5, w - r - 5)), int(rng.integers(r + 5, h - r - 5))
        if any((x - a) ** 2 + (y - b) ** 2 < (r + c + 8) ** 2 for a, b, c in taken):
            continue
        taken.append((x, y, r))
        red = int(rng.integers(150, 225))
        cv2.circle(img, (x, y), r, (35, 40, red), -1, cv2.LINE_AA)
        cv2.circle(img, (x - r // 3, y - r // 3), r // 4, (90, 90, min(255, red + 30)), -1, cv2.LINE_AA)
        boxes.append((x - r, y - r, x + r, y + r))
    for x, y, r in taken[:bruises]:
        cv2.ellipse(img, (x + r // 3, y + r // 4), (r // 3, r // 4), 20, 0, 360, (25, 45, 70), -1, cv2.LINE_AA)
        spots.append((x, y, r))
    img = cv2.GaussianBlur(img, (3, 3), 0)
    return np.clip(img * light, 0, 255).astype(np.uint8), boxes, spots

Also uses patch.py

import numpy as np
import cv2
import torch

def make(n, seed=0, light=1.0, bruise_rate=0.5):
    """Simulated apple photos. Label 1 means bruised."""
    rng = np.random.default_rng(seed)
    X, y = [], []
    for _ in range(n):
        img = np.full((48, 48, 3), (70, 75, 80), np.uint8)
        c = tuple(int(v) for v in rng.integers(19, 30, 2))
        r = int(rng.integers(15, 20))
        cv2.circle(img, c, r, (35, 40, int(rng.integers(150, 225))), -1, cv2.LINE_AA)
        bad = rng.random() < bruise_rate
        if bad:
            at = (c[0] + int(rng.integers(-8, 9)), c[1] + int(rng.integers(-8, 9)))
            cv2.circle(img, at, int(rng.integers(3, 6)), (25, 45, 70), -1, cv2.LINE_AA)
        img = img * light + rng.normal(0, 8, img.shape)
        X.append(img.clip(0, 255) / 255)
        y.append(int(bad))
    X = torch.tensor(np.array(X), dtype=torch.float32).permute(0, 3, 1, 2)
    return X, torch.tensor(y)

def net():
    return torch.nn.Sequential(
        torch.nn.Conv2d(3, 16, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
        torch.nn.Conv2d(16, 32, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
        torch.nn.Flatten(), torch.nn.Linear(32 * 12 * 12, 2))

def train(model, X, y, epochs=8):
    opt = torch.optim.Adam(model.parameters(), lr=2e-3)
    for _ in range(epochs):
        for i in torch.randperm(len(X)).split(64):
            loss = torch.nn.functional.cross_entropy(model(X[i]), y[i])
            opt.zero_grad(); loss.backward(); opt.step()
    return model

def accuracy(model, X, y):
    return (model(X).argmax(1) == y).float().mean().item()

Vision model or your own?

So when should you use each? A general model is great for prototypes, for rare questions, for reading documents, and for working out what's possible. Your own trained model wins on speed, cost at high volume, working offline, privacy, and precise locations. A common pattern is to use the general model to label data, and to use that to train a small, fast model.

Need	Use
Quick prototype	General vision model
Rare, varied questions	General vision model
Millisecond speed	Your own model
High volume	Your own model
Exact locations	Your own model
Labelling help	General model, then train

Summary

General models are a great first step, not always the last. Next, let's see vision beyond the factory.

Key takeaways

Real models can count well
Three images isn't proof
Speed and cost decide

Next up: Vision from Space — seeing plant health

❮ Image Search Vision from Space ❯