Precision/Recall Tuning & Reporting
The discipline of measuring and sequencing precision vs. recall when tuning AI detectors/reviewers/triage agents, and the recurring pattern of articles reporting only one side while omitting the other.
Also known as: precision before recall, precision-recall tradeoff, false-positive rate, false-negative rate, recall-null, calibration (precision/recall/variance), confusion matrix
How authors use it
Oldest first (extraction date when no publication date is known).
| Author | Their term | Their meaning |
|---|---|---|
| How Figma Uses AI Agents for Security Renato Losio · 2026-09-06 | precision before recall | Sullivan and Girardeau's tuning-order principle: fix an agent's false-positive rate (precision) before trying to improve its false-negative rate / coverage (recall), because a team's inventory of already-known historical bugs can only be used to measure recall, not precision. |
Documents (1)
- How Figma Uses AI Agents for Security Renato Losio · 2026-09-06 source ↗ Reader 🖥️
Figma's security engineering team, as relayed by InfoQ from the company's own blog posts, holds that AI agents equipped with persistent, separated memory, tightly scoped tools, and strict human-review guardrails (draft-by-default PRs, anti-leak prompts) can absorb much of a security team's repetitive investigative and detection work -- reporting roughly 70% faster complex-alert resolution, 20% fewer on-call pages, 100+ newly found vulnerabilities, and 80% code-reviewer precision -- while insisting that the right balance between agent autonomy and human oversight remains an open, evolving question rather than a solved one.
Propositions touching this concept (5)
By publication month, oldest first (extraction month when no publication date is known).
2026-05
- explicitdevelopedFalse-positive rate from security LLMs is dominated by two factors: programming-language memory safety (C/C++ produce more false positives than Rust) and model hedging bias.
2026-09
- explicitThe main stated lesson is to improve precision before recall, because the historical bugs a team already has can only measure recall and barely help with the precision that must be fixed first.
- explicitGoogle built Mantis specifically to address the high false-positive rate and hallucinated vulnerabilities produced by conventional AI-powered code scanning, whose true-positive rate falls under 7%.
- implicitThe article itself never states any quantified accuracy figure for Mantis's own output (its true-positive or false-positive rate after the review/critic stages) — only the problem baseline it addresses (<7% true-positive rate for 'conventional' scanning) is quantified.
- explicitThe recall-null / attribution-gain pattern replicates when large language models, rather than humans, occupy the reviewer role across ninety machine reviews and three models, with attribution collapsing to exactly 0.00 without a baseline.
Open gaps (5)
- incompleteCross-family review is recommended from repeated anecdote ('it really does work'), with no false-positive rate, no cost/benefit accounting, and no discussion of when it fails to catch things or produces noise that costs more review time than it saves.Have your agent record video demos of its work with shot-scraper video — read next: See the GitHub Copilot 'Rubber Duck' analysis of cross-family review as cost-effective only when invoked sparingly at high-signal moments, and eval methodology for measuring reviewer models.
- incompleteNo systematic false-positive/false-negative rate is reported for the J-lens as a safety-monitoring tool. The three deception/misalignment case studies (blackmail scenario, benchmark fabrication, sabotage model organism) are existence proofs on curated examples, not a validated detector with a measured base rate of J-space 'suspicious' signatures in ordinary benign use, nor a measured miss rate against genuinely deceptive behavior the method fails to catch.A global workspace in language models — read next: Check the full paper (transformer-circuits.pub) or Neel Nanda's invited commentary/independent replication for any quantitative detection statistics.
- incompleteNo evidence (adoption numbers, incident-rate change, false-positive rate) is offered that this preview actually improves release safety or speed — the piece is a product announcement, not a study.AWS Expands DevOps Agent with AI-Powered Release Management to Validate Code Before Production — read next: Case studies or retrospectives after general availability.
- incompleteNo metrics are given anywhere in the piece — no false-positive rate, no override/dismissal rate, no measured change in reviewer behavior or time-to-catch for security issues — after deploying Dash+MCP in code review. Every claim about avoiding 'a false sense of security correctness' and about developer trust is a stated design principle, never an evaluated outcome.Dropbox Integrates MCP and Dash to Close the Gap Between Security Design and Code Review — read next: The linked Dropbox engineering blog post (dropbox.tech/security/dropbox-mcp-dash-design-code-security) or a follow-up post, if one exists, with adoption/precision data.
- incompleteThe diff classifier that separates 'cosmetic' from 'material' manifest changes is presented as the solution to alert fatigue and rubber-stamping, but its own false-negative rate — a material, malicious redefinition misclassified as cosmetic and auto-approved without ever reaching human review — is never examined, even though this is exactly the failure mode that would silently defeat manifest pinning's central promise.Article: Securing MCP in Production: Defense-in-Depth Beyond the Gateway — read next: Literature on classifier-gated security controls and their own adversarial robustness; compare against the structurally identical anomaly-gating false-negative problem already flagged elsewhere in this corpus for a pre-filter in front of an LLM-driven SOC pipeline.