Home › Accuracy
How accurate is Qorami?
A guardrail you can't measure is a guardrail you can't trust. So we publish the numbers — on a labeled benchmark, reproducible, and honest about the limits.
0
dangerous misses — no risky email scored "send"
100%
agreement with operator labels (92/92)
92
labeled emails · 5 languages · 4 risk profiles
Per-class results
| Class | Precision | Recall | Emails |
|---|---|---|---|
| Authorize (send) | 100% | 100% | 32 |
| Ask a human | 100% | 100% | 27 |
| Block | 100% | 100% | 33 |
The number that matters most: 0 dangerous misses. Every email that should be
blocked or reviewed is — none is ever silently sent. The set deliberately loads in the hard,
dangerous cases (leaked API keys and plaintext passwords across five languages, prompt-injection /
agent-hijack attempts, payment-card and SSN leaks) precisely because those are the ones that must
never slip through. Qorami fails toward a human, not toward a send.
What this measures (and doesn't)
- This is the deterministic rule engine — the offline floor, tuned for near-zero false positives. In production a semantic LLM judge (Claude) runs on top to add recall on what rules can't catch alone: paraphrased secrets, implied commitments and subtle policy-sensitive intent.
- The set is 92 labeled emails in FR / EN / ES / DE / IT across sales, support, legal/finance, HR, ops and IT — the three decisions and four risk profiles — including adversarial cases: prompt-injection / agent-hijack attempts, leaked DB connection strings and plaintext credentials, and near-miss benign phrasing ("please ignore my previous email", "a password reset is required") that must not trip a detector.
- "Precision" = of the emails we flagged for a class, how many belonged there. "Recall" = of the emails in a class, how many we caught.
- It's a curated benchmark, not live traffic. Real inboxes are messier and adversaries adapt — 100% here means the floor is sound on the cases we can label, not a promise of 100% in the wild. That's exactly why every risky-looking email still fails toward a human.
Reproduce it
The benchmark and scorer ship in the repo. Run:
npm run evalEvery detection change re-runs this set, so accuracy can only move with eyes on it.