# Penthera and 25 free security tools, on four public benchmarks

> How does Penthera rank against the free tools, on public data other people labelled?

Measured on 2026-10-25, on the sealed half of each dataset, scored once. Sealed test half, scored once (E11b § 10).

- [Leaked secrets, Samsung CredData](https://penthera.org/benchmarks/secrets.md): Penthera 2.0.0 ranked 2nd of 10 free tools by F1 (0.74), behind deepsecrets (0.78).
- [Malicious npm packages, DataDog's dataset](https://penthera.org/benchmarks/npm-malware.md): Penthera 2.0.0 ranked 1st of 6 free tools by caught minus false alarms (0.71); next, malcontent (0.63).
- [Malicious PyPI packages, DataDog's dataset](https://penthera.org/benchmarks/pypi-malware.md): Penthera 2.0.0 ranked 1st of 8 free tools by caught minus false alarms (0.82); next, GuardDog (0.74).
- [Injection bugs in JavaScript libraries, SecBench.js](https://penthera.org/benchmarks/js-injection.md): Penthera 2.0.0 ranked 1st of 8 free tools by share found in the right file (0.70), within the margin of error of odgen (0.68).

The free tools ran on their defaults. Penthera was tuned on the other half of the dataset, then scored once on this half. A benchmark is not your code. Accuracy on real projects is experiment E1, pre-registered and not yet run. Paid tools (GitGuardian, Socket, Snyk, Phylum, Semgrep Pro) are not here. None publishes results on these benchmarks; we cannot measure them (cost and licence terms).

- Numbers: https://github.com/danoszz/penthera/blob/main/bench/public/published.json
- Protocol: https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E11b-improvement-study.md