What we measure, what we do not, and why

Each case is a small project holding the smallest code that exhibits, or safely avoids, one vulnerability. Each carries a comment saying what is wrong, what is expected and why. The corpus is versioned: a change to it is a change to the published figures.

The expected result is a judgement about the code, decided before the engine runs, never a description of what the engine produced. When the two disagree we investigate the engine first. A label written to match engine output measures nothing, and we have had to correct labels that had drifted that way.

Every case is analysed the way a customer scan runs: the full orchestrator at maximum depth, no external scanners, no model in the loop. The figures describe the engine that ships identically everywhere, not a laboratory configuration assembled for the occasion.

A third of the corpus is code that is not vulnerable, most of it the corrected form of a case that is. A corpus of vulnerabilities alone cannot detect a false positive, and false positives are what make a security tool stop being read.

A factual claim: these assets exist and these relations hold between them. Measurable, and the first thing we intend to measure. It needs a corpus of complete repositories annotated by hand, two independent reviewers, and the agreement between them published next to the score. Anything less measures the annotation process rather than the extractor.

Factual, but only once someone states what the attacker starts with. “An attacker can reach the database” is neither true nor false until you say whether they begin as an anonymous visitor, an authenticated user, or a compromised build runner. The same engine is right under one assumption and wrong under another, so a figure without that assumption printed beside it is not reproducible.

A claim about what matters more, and there is no fact of the matter independent of an objective. A pre-revenue startup and a regulated bank read the same repository and are both right to rank it differently. Any correct order we published would be our own product opinion, validated against a ground truth we also wrote.

Act now, plan, verify: a recommendation against a policy, and the policy is a product decision. We can test that it follows its own policy, that no finding is silently dropped, and that every recommendation cites evidence that exists. That is a conformance test. Calling it accuracy would invite exactly the misreading this page is written to prevent.

Each layer shows how many findings its precision was computed from. The layers with small samples are the recently added ones, and their figures will move as their corpus grows. This cuts against the flattering rows as much as the awkward one: a perfect score over nine findings is a weak claim, not a strong one.

Findings the engine produces on code that is not vulnerable are listed with the analyzer at fault and the reason the finding is wrong. They are counted against precision, never excluded from it. Three of them fire on the mitigation the rule itself recommends, which is precisely the kind of defect a benchmark exists to surface rather than absorb.

It reads an application's own integration code for LLM and agent mistakes: model output reaching a shell, a credential placed in a prompt, an agent tool that can run anything. It is not a measure of an AI inside Cybseco. This benchmark runs the deterministic engine with no model in the loop.

A vulnerability that is real and outside the engine's current reach is recorded as an accepted gap, excluded from recall, and listed. Recording a limit is how a future improvement shows up as a measured gain instead of being claimed today.