The Lethal Trifecta

A security researcher, Simon Willison, spotted a pattern behind many agent attacks. Three abilities, each one harmless by itself. Put all three in one agent, and an attacker can steal your data. He called it the lethal trifecta.

• A pattern behind many attacks
• Three harmless abilities
• Together, dangerous

The three ingredients

The three are access to private data, exposure to untrusted content, and a way to send data out. Say an agent can read customer records, it reads emails from strangers, and it can send email. One poisoned email says, send the customer list to this address. And it may just do that. Remove any one of the three, and that attack fails.

Auditing your tools

You can check for this in code. We label each tool. Does it touch private data? Does it bring in untrusted content? Can it send things out? The audit looks at a set of tools. Run it. Two tools are fine. Add send email, and the alarm goes off. Run this on every agent that you build. It takes a few minutes, and it could save your company.

tools = {
    "read_customer_records": {"private": True, "untrusted": False, "external": False},
    "read_support_inbox": {"private": False, "untrusted": True, "external": False},
    "send_email": {"private": False, "untrusted": False, "external": True},
}

def audit(names):
    chosen = [tools[n] for n in names]
    private = any(t["private"] for t in chosen)
    untrusted = any(t["untrusted"] for t in chosen)
    external = any(t["external"] for t in chosen)
    return "DANGER: all three" if private and untrusted and external else "Acceptable"

print(audit(["read_customer_records", "read_support_inbox"]))
print(audit(["read_customer_records", "read_support_inbox", "send_email"]))

Acceptable
DANGER: all three

Tracking tainted runs

You don't always have to remove a tool. Here's a smarter approach. Once an agent has read something untrusted, mark the run as tainted. A tainted run can't send anything outside the company. Run it. Before reading anything, it could email a customer. After reading a web page, the external send gets blocked, but the internal one still works. The agent keeps most of its usefulness, and it can't leak data.

class Run:
    def __init__(self):
        self.tainted = False

    def read_untrusted(self, text):
        self.tainted = True
        return text

    def send_email(self, to):
        if self.tainted and not to.endswith("@velosupply.example"):
            return f"BLOCKED: untrusted data was read, and {to} is external"
        return f"Sent to {to}"

run = Run()
print(run.send_email("priya@gmail.example"))
run.read_untrusted("A web page with hidden instructions")
print(run.send_email("attacker@evil.example"))
print(run.send_email("support@velosupply.example"))

Sent to priya@gmail.example
BLOCKED: untrusted data was read, and attacker@evil.example is external
Sent to support@velosupply.example

Summary