How we benchmarked Penthera against 25 free tools, losses included

· Daan van der Zwaag

How do you benchmark your own security tool against other tools without fooling yourself?

On 25 October 2026 we published four rankings. Each puts Penthera next to every free, open-source tool we could run offline on one public dataset, scored the same way, on a half of the dataset we sealed before any tuning. Penthera ranked 1st of 6 on malicious npm packages and 1st of 8 on malicious PyPI packages. On injection bugs in JavaScript libraries it ranked 1st of 8, within the margin of error of ODGen. On leaked secrets it ranked 2nd of 10, behind deepsecrets.

This post is about how we got those numbers and how far you should trust them. Every tool, interval and limit is on the benchmark pages.

Why a sealed half

The easy way to improve a scanner on a benchmark is to look at what it misses and write rules for those cases. The score goes up. It also means less, because the rules now know the answers.

So we split each dataset in half before anyone looked at a single error. The split is seeded (seed 20261024), stratified and 50/50. It was committed as a list of ids and hashes before either half was scored. We tuned on the dev half only. The test half was scored once, at the end, and only its numbers are published. The runner refuses to score the test half without a flag that says it is the final run.

Rules written before the run

Before the test half was read, we registered how each benchmark ranks:

A rank needs at least five other free tools measured on the same half. A tool trained on the dataset is shown and not ranked. That rule took out CredSweeper, which scored F1 0.99: it is made by the CredData authors, and its model was trained on CredData. A lookup of already-reported packages is shown and not ranked too, so osv-scanner is not in the malware ranks.

What a change could be based on

Every improvement had to be a general pattern, stated in words, such as "YAML values are unquoted". Never a file name, a package name or a value from a dataset. Each change got unit tests from our own fixtures, not from dataset content. After tuning, a reviewer who did not write the rules checked every rule and test added since 1.9.0. They found no dataset identifiers, values or package names, and no added indicator of compromise.

What we found

For scale: Penthera 1.9.0, before the study, scored 0.18 on secrets and 0.16 on SecBench.js on the same test half.

The drop from dev to test

On the dev half, after tuning, Penthera scored higher on every benchmark than it then did on the sealed half: secrets F1 0.81 to 0.74, SecBench.js 0.88 to 0.70, npm 0.83 to 0.71, PyPI 0.92 to 0.82. That gap is what tuning on the data you score looks like. It is the reason for the split.

Two test runs, both reported

The first final run was stopped before anyone read it, when we decided on one more round of tuning. Its outputs were packed into one archive, its hash recorded and every readable copy deleted. After round two we opened it and published both runs, so neither was picked after the fact. Secrets and npm scored the same in both. PyPI went from 0.75 to 0.82 and SecBench.js from 0.63 to 0.70. Penthera 1.9.0 scored the same in both runs, so the harness is deterministic.

The leaks the audit found

The same review found two leaks in the split itself:

A clean measure needs new data: malware samples published after the freeze, and real projects.

What this does not show

Check it yourself

Sources: published.json, RESULTS.md, E11b-improvement-study.md.

penthera.org · Benchmarks · Blog · Live check · Terms · Trust · Source · MIT license