Benchmark Methodology

What this benchmark measures

Every vector in lib/fuzzer/attacks is a known attack. Running them through the safety engine measures its recall: of 5,023 attacks we know are malicious, how many does the deterministic engine block? The misses are honest findings about our own product, not hidden failures.

Corpus composition

• Source: lib/fuzzer/attacks — 5,023 vectors across 32 categories (getStats().totalAttacks is the canonical count).
• Categories span jailbreak, prompt-injection, system-leak, data-exfil, encoding, multilingual, tool-abuse, memory-poison, multi-agent, RAG-poison, social-engineering, harmful-content, bias, hallucination, and more.
• Vectors are generated by expanding base patterns through transforms (translation, encoding, roleplay, hypothetical framing, etc.), which is why the corpus is deliberately adversarial and includes many semantic-framing attacks.

Scoring rules

• Engine: runSafetyChecks() from lib/safety.ts, run in STRICT mode.
• A vector counts as caught if the result is blocked === true or allowed === false.
• Deterministic and order-stable: no randomness, no sampling — the same inputs produce the same catch count on any machine.
• In-process: imports lib/safety + lib/fuzzer/attacks directly. No server, no network, no model call, no spend.
• Latency is per-call wall-clock for the deterministic layers only — it excludes network transport and any optional LLM call.

Current result (2026-07-08, git 8f3d1c8a)

The full machine-readable artifact — including the per-category breakdown and the exact engine layer list — is published at /evidence/fuzzer-corpus-score.json.

Catch rate by category

Categories at or near 0% (e.g. multilingual, social-engineering, bias) are the semantic-framing attacks the keyword engine is not designed to catch on its own — they are exactly what the LLM verdict layer exists to handle. We publish them rather than hide them.

Reproduce it yourself

The score is regenerated by a single deterministic script — no key, no spend:

# from a clone of the repo
npm ci
npm run eval:fuzzer-corpus
# → writes public/evidence/fuzzer-corpus-score.json

To verify the receipts a real decision produces — independently of our site — use the open verification packages:

# TypeScript verifier
npm install @omegaengine/verify

# Python auditor
pip install omega-audit

• The verdict engine (precision/recall against human labels) is reproduced separately with npm run eval:judge — see the Judge Card.
• Every published number here is either deterministic (this page) or carries its budget/seed-set caveat (the Judge Card). No number is sourced from a failing benchmark run.

Related evidence