Aspect Sentiment
What exactly did they like?.
Three stars. That tells you almost nothing. The customer might have loved the book, and hated the delivery. If you average those into one number, then you've thrown away the only thing that you could've acted on. The real question is, what exactly did they like, and what did they dislike?
- Three stars tells you nothing
- Loved the book, hated the delivery
- What exactly did they like?
Opinions as data
With a schema, this is straightforward. Each opinion has an aspect, from a fixed list, and a feeling. And the answer is a list, as a single review can hold several. Here's a mixed review. Run it. Three opinions from one sentence. The book's quality is positive. Delivery is negative. Price is negative. Run that across a month of reviews, and you can count them up. It might show that eighty percent of complaints are about delivery, rather than books. That's something a manager can act on, and a star rating could never tell you that.
from typing import Literal
from pydantic import BaseModel
from llm import ask
class Opinion(BaseModel):
aspect: Literal["delivery", "price", "book quality", "service"]
feeling: Literal["positive", "negative"]
class Result(BaseModel):
opinions: list[Opinion]
review = "The novel itself is beautiful, but it came two weeks late and costs far too much."
for o in ask(f"List every opinion in this review:\n{review}", schema=Result).opinions:
print(f"{o.aspect:<14} {o.feeling}")book quality positive delivery negative price negative
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Also uses llm.py
from google import genai
from google.genai import types
client = genai.Client()
def ask(prompt, system=None, schema=None, temperature=0.1):
config = types.GenerateContentConfig(temperature=temperature, system_instruction=system)
if schema:
config.response_mime_type = "application/json"
config.response_schema = schema
r = client.models.generate_content(model="gemini-flash-latest", config=config, contents=prompt)
return r.parsed if schema else r.textDesigning the categories
Keep the list of aspects short, and let the business choose them. Read a hundred reviews first, to see what people actually talk about. Add an other category, and review it each month, to catch new topics. And check a sample by hand, every so often, as models drift when customers start using new words.
Summary
Split opinions by topic, and they become actionable. Next, what do you do with thousands of them?
Key takeaways- Stars hide the detail
- One review, many opinions
- Use a short category list