Loading...
Loading...
How well do AI agents resist agentic attacks — the ones that live in tool calls, not prose? We score what an agent does, not what it says. Verdicts are observational— the scorer inspects the actual tool calls, so "did it call send_email(attacker@…)?" is a fact, not an opinion. Re-run it and check.
Deterministic and reproducible by anyone. The ladder is the point: prompt tricks help, but only enforcement outside the model resists the whole corpus.
Point-in-time — models are non-deterministic, so numbers drift run-to-run. Frontier models refuse direct asks but still fall to indirect prompt injection; the gap between 96% and a breach is one attack.
50 attacks grounded in published taxonomies — defensible provenance, not invention.
Methodology: verdicts come from lib/redteam/agentTrace.ts (observational, deterministic). Archetype rows are reproducible to the digit; live rows are measured against real models on the date shown. The corpus (lib/redteam/agentThreatCorpus.ts) is small and author-built but taxonomy-grounded — a larger, independently-reviewed corpus is on the roadmap. We'd rather under-claim and earn trust.