# Free PyPI malware detectors on DataDog's dataset: 8 tools ranked

> Which free tool catches the most malicious PyPI packages, with the fewest false alarms?

On a sealed half of DataDog's malicious PyPI packages (150 with a payload) and 150 popular benign packages, Penthera 2.0.0 ranked 1st of 8 free tools, with caught minus false alarms 0.82. Next was GuardDog with 0.74. Measured on 25 October 2026.

## Read this before the numbers

- GuardDog's authors helped build this dataset, so GuardDog's catch rate is inflated by construction.
- osv-scanner looks packages up in a list of reports. It is shown, not ranked: a lookup of known packages is not detection.
- Penthera missed its own pre-registered target for false alarms on this benchmark.
- The free tools ran on their defaults. Penthera was tuned on the other half of the dataset, then scored once on this half.
- A benchmark is not your code. Accuracy on real projects is experiment E1, pre-registered and not yet run.

## Every tool, with 95% intervals

Scored by caught minus false alarms on benign packages: Youden's J. Counts are found over total; intervals are 95% Wilson.

| Rank | Tool | Caught | False alarms | caught minus false alarms |
|---|---|---|---|---|
| 1 | **Penthera 2.0.0** | 0.87 [0.81, 0.92] (131/150) | 0.05 [0.03, 0.10] (8/150) | 0.82 |
| 2 | GuardDog 3.2.0 | 0.85 [0.79, 0.90] (128/150) | 0.11 [0.07, 0.17] (16/147) | 0.74 |
| 3 | malcontent 1.27.1 (bf280ad6), yara-x 1.21.0 | 0.93 [0.87, 0.96] (139/150) | 0.22 [0.16, 0.29] (32/147) | 0.71 |
| 4 | yara yara-python 4.5.4 (YARA 4.5), YARA-Forge core 20261004 | 0.25 [0.18, 0.32] (37/150) | 0.00 [0.00, 0.03] (0/147) | 0.25 |
| 5 | clamav 1.5.4 | 0.00 [0.00, 0.02] (0/150) | 0.00 [0.00, 0.03] (0/147) | 0.00 |
| 6 | ossgadget 0.2.20 (414b6a59) | 0.77 [0.70, 0.83] (116/150) | 0.97 [0.93, 0.99] (143/147) | -0.20 |
| 7 | aura 2.1 (SourceCode-AI/aura b40e6dd2) | 0.59 [0.51, 0.67] (88/149) | 0.83 [0.75, 0.88] (100/121) | -0.24 |
| 8 | bandit4mal lyvd/bandit4mal 03ba95f3 (bandit 1.6.3 fork) | 0.59 [0.51, 0.67] (89/150) | 0.84 [0.78, 0.89] (124/147) | -0.25 |
| – | Penthera 1.9.0 | 0.83 [0.77, 0.88] (125/150) | 0.09 [0.06, 0.15] (14/150) | 0.74 |
| – | osv-scanner 2.6.0 | 0.74 [0.66, 0.80] (110/149) | 0.00 [0.00, 0.03] (0/147) | 0.74 |

Not ranked: Penthera 1.9.0 (the previous release, for comparison); osv-scanner 2.6.0 (lookup of already-reported packages, not detection).

## Where another tool does better than Penthera

- Caught: malcontent 0.93, Penthera 0.87. Overall, malcontent scored 0.71.
- False alarms: yara 0.00, Penthera 0.05. Overall, yara scored 0.25.

## Sources

- Dataset: [DataDog malicious software packages dataset](https://github.com/DataDog/malicious-software-packages-dataset)
- Numbers: https://github.com/danoszz/penthera/blob/main/bench/public/published.json (measured 2026-10-25; Sealed test half, scored once (E11b § 10).)
- Protocol: https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E11b-improvement-study.md
- All four benchmarks: https://penthera.org/benchmarks.md