Free PyPI malware detectors on DataDog's dataset: 8 tools ranked
On a sealed half of DataDog's malicious PyPI packages (150 with a payload) and 150 popular benign packages, Penthera 2.0.0 ranked 1st of 8 free tools, with caught minus false alarms 0.82. Next was GuardDog with 0.74. Measured on 25 October 2026.
Which free tool catches the most malicious PyPI packages, with the fewest false alarms?
Penthera 2.0.0 ranked 1st of 8 free tools by caught minus false alarms (0.82); next, GuardDog (0.74). Scored by caught minus false alarms on benign packages: Youden's J.
- Penthera0.82
- GuardDog0.74
- malcontent0.71
- yara0.25
- clamav0.00
- ossgadget-0.20
- aura-0.24
- bandit4mal-0.25
Every tool, with 95% intervals
Counts are found over total. Intervals are 95% Wilson intervals. Penthera's row is bold.
| Rank | Tool | Caught | False alarms | Score: caught minus false alarms |
|---|---|---|---|---|
| 1 | Penthera 2.0.0 | 0.87 [0.81, 0.92] 131/150 | 0.05 [0.03, 0.10] 8/150 | 0.82 |
| 2 | GuardDog 3.2.0 | 0.85 [0.79, 0.90] 128/150 | 0.11 [0.07, 0.17] 16/147 | 0.74 |
| 3 | malcontent 1.27.1 (bf280ad6), yara-x 1.21.0 | 0.93 [0.87, 0.96] 139/150 | 0.22 [0.16, 0.29] 32/147 | 0.71 |
| 4 | yara yara-python 4.5.4 (YARA 4.5), YARA-Forge core 20261004 | 0.25 [0.18, 0.32] 37/150 | 0.00 [0.00, 0.03] 0/147 | 0.25 |
| 5 | clamav 1.5.4 | 0.00 [0.00, 0.02] 0/150 | 0.00 [0.00, 0.03] 0/147 | 0.00 |
| 6 | ossgadget 0.2.20 (414b6a59) | 0.77 [0.70, 0.83] 116/150 | 0.97 [0.93, 0.99] 143/147 | -0.20 |
| 7 | aura 2.1 (SourceCode-AI/aura b40e6dd2) | 0.59 [0.51, 0.67] 88/149 | 0.83 [0.75, 0.88] 100/121 | -0.24 |
| 8 | bandit4mal lyvd/bandit4mal 03ba95f3 (bandit 1.6.3 fork) | 0.59 [0.51, 0.67] 89/150 | 0.84 [0.78, 0.89] 124/147 | -0.25 |
| Not ranked | Penthera 1.9.0 | 0.83 [0.77, 0.88] 125/150 | 0.09 [0.06, 0.15] 14/150 | 0.74 |
| Not ranked | osv-scanner 2.6.0 | 0.74 [0.66, 0.80] 110/149 | 0.00 [0.00, 0.03] 0/147 | 0.74 |
- GuardDog: found most samples in this dataset, so its detection rate is inflated by construction.
- A rank needs at least five other free tools measured; this benchmark has 7.
- Median score of the other free tools: 0.00.
Not ranked, and why
- Penthera 1.9.0: the previous release, for comparison. Score 0.74.
- osv-scanner 2.6.0: lookup of already-reported packages, not detection. Score 0.74.
Where another tool does better than Penthera
- Caught: malcontent 0.93, Penthera 0.87. Overall, malcontent scored 0.71.
- False alarms: yara 0.00, Penthera 0.05. Overall, yara scored 0.25.
Tools that are not here
Paid tools (GitGuardian, Socket, Snyk, Phylum, Semgrep Pro) are not here. None publishes results on these benchmarks; we cannot measure them (cost and licence terms).
Published elsewhere, on other data
Not comparable with the numbers above: other datasets, other scoring.
Check it yourself
The dataset is DataDog malicious software packages dataset. The ranking rules were written down before the run: the protocol. Every number on this page is read from published.json when the site is built; the results file has the method and per-family detail, and the runner's README says how to reproduce it.
Measured on . Sealed test half, scored once (E11b § 10). All four benchmarks.
penthera.org · Benchmarks · Blog · Live check · Terms · Trust · Source · MIT license