Asking Models About Images
Multimodal language models.
Today, you can show a photo to a language model, and just ask what's wrong with it. It reads labels, describes scenes, and reasons about what it sees. For many tasks, this replaces months of model training. But it's not always the right answer.
- Show a photo, ask a question
- Replaces months of training
- But not always right
Asking about a photo
We send an image along with a question. Run it. It describes the photo in natural language, and it even suggests a flaw that an inspector might note. It's a real call to a real model. No training, no dataset. You can build a working prototype in an afternoon.
import io
from PIL import Image
from google import genai
from google.genai import types
from skimage import data
buf = io.BytesIO()
Image.fromarray(data.coffee()).save(buf, "JPEG")
client = genai.Client()
reply = client.models.generate_content(
model="gemini-flash-latest",
contents=[types.Part.from_bytes(data=buf.getvalue(), mime_type="image/jpeg"),
"In one sentence, what is in this image? Then one possible flaw a quality inspector might note."])
print(reply.text.strip())**Description:** A single shot of espresso with a golden-brown crema is served in a reddish-brown ceramic cup and saucer, accompanied by a small metal spoon on a wooden tabletop. **Possible Flaw:** A quality inspector might note coffee splatters and residue on the inner white ceramic rim above the crema line, indicating a messy pour.
Structured inspection
Let's use this properly, with a schema. We draw a simple diagram of a bruised apple, and we ask for a grade, in fixed fields. Run it. Clean JSON, with a grade and a reason. You could send that straight to a sorting machine. But hold on, is this the right design for a line handling ten apples per second? Let's think about that.
import io
import json
from PIL import Image, ImageDraw
from google import genai
from google.genai import types
from pydantic import BaseModel
class Grade(BaseModel):
fruit: str
bruised: bool
grade: str
reason: str
img = Image.new("RGB", (320, 320), (40, 40, 40))
d = ImageDraw.Draw(img)
d.ellipse((50, 50, 270, 270), fill=(200, 30, 30))
d.ellipse((170, 90, 230, 140), fill=(90, 50, 30))
buf = io.BytesIO(); img.save(buf, "JPEG")
client = genai.Client()
r = client.models.generate_content(
model="gemini-flash-latest",
contents=[types.Part.from_bytes(data=buf.getvalue(), mime_type="image/jpeg"),
"This is a simple diagram of an apple. Grade it A, B, or reject."],
config=types.GenerateContentConfig(response_mime_type="application/json", response_schema=Grade))
print(json.dumps(json.loads(r.text), indent=1)){
"fruit": "apple",
"bruised": true,
"grade": "B",
"reason": "The apple has a visible brown bruise on its surface, which prevents it from meeting Grade A standards but remains acceptable as Grade B."
}A big model, or a small trained one?
Here's how to choose. A multimodal language model needs no data, handles anything, and it's great for prototypes, low volume, and rare cases. A small trained model costs almost nothing per image, it runs in milliseconds on the device, and it works offline. For AppleEye, with ten apples a second, we need the small model. A multimodal model could review the hard cases.
| LLM vision | Trained small model | |
|---|---|---|
| Data needed | None | Hundreds of images |
| Speed | Seconds | Milliseconds |
| Cost per image | Fractions of a cent | Almost zero |
| Offline | No | Yes |
| Best for | Prototypes, rare cases | High volume |
Summary
Multimodal models are amazing for prototypes, and for rare hard cases. Small trained models win at volume. Next, we go from understanding images to creating them.
Key takeaways- Ask questions about images
- Use a schema for clean results
- Small trained models win at volume