# Free secret scanners on Samsung CredData: 10 tools ranked

> Which free secret scanner finds the most leaked secrets on CredData?

On the sealed test half of Samsung CredData (2,220 labelled secrets), deepsecrets ranked 1st of 10 free tools, with F1 0.78. Penthera 2.0.0 ranked 2nd with 0.74: better precision (0.89 against 0.80), worse recall (0.64 against 0.77). Measured on 25 October 2026.

## Read this before the numbers

- CredData replaces each real secret with random characters of the same shape. That favours pattern rules over tools that check a key with its provider.
- trufflehog ran with verification off: verifying calls each provider's API, and every tool ran offline. Checking a key with its provider is its main feature, so its score here undersells it.
- CredData's labels lean partly on other scanners' output, and it is split by file, so most repositories have files on both halves.
- The free tools ran on their defaults. Penthera was tuned on the other half of the dataset, then scored once on this half.
- A benchmark is not your code. Accuracy on real projects is experiment E1, pre-registered and not yet run.

## Every tool, with 95% intervals

Scored by F1 of strict precision and in-scope recall, as the CLI reports. Counts are found over total; intervals are 95% Wilson.

| Rank | Tool | Precision | Recall | F1 |
|---|---|---|---|---|
| 1 | deepsecrets 2.2.0 | 0.80 [0.79, 0.82] (2,752/3,427) | 0.77 [0.75, 0.78] (1,704/2,220) | 0.78 |
| 2 | **Penthera 2.0.0** | 0.89 [0.87, 0.90] (1,640/1,849) | 0.64 [0.62, 0.66] (1,415/2,220) | 0.74 |
| 3 | noseyparker 0.24.0 | 0.76 [0.74, 0.78] (1,898/2,501) | 0.62 [0.60, 0.64] (1,387/2,220) | 0.69 |
| 4 | gitleaks 8.30.1 | 0.85 [0.84, 0.86] (2,242/2,635) | 0.37 [0.35, 0.39] (831/2,220) | 0.52 |
| 5 | ripsecrets 0.1.11 | 0.65 [0.64, 0.67] (1,883/2,877) | 0.35 [0.33, 0.37] (766/2,220) | 0.45 |
| 6 | kingfisher 2.14.0 | 0.96 [0.94, 0.97] (797/834) | 0.29 [0.28, 0.31] (653/2,220) | 0.45 |
| 7 | detect-secrets 1.5.0 | 0.30 [0.29, 0.31] (2,237/7,444) | 0.53 [0.51, 0.56] (1,187/2,220) | 0.38 |
| 8 | secretlint 13.0.7 | 0.92 [0.89, 0.94] (459/499) | 0.19 [0.18, 0.21] (424/2,220) | 0.32 |
| 9 | whispers 2.4.0 | 0.46 [0.43, 0.50] (281/605) | 0.09 [0.08, 0.10] (193/2,220) | 0.15 |
| 10 | trufflehog 3.99.3 | 0.40 [0.35, 0.45] (152/383) | 0.05 [0.05, 0.07] (122/2,220) | 0.10 |
| – | Penthera 1.9.0 | 0.56 [0.51, 0.60] (237/425) | 0.10 [0.09, 0.12] (233/2,220) | 0.18 |
| – | credsweeper 1.19.4 | 1.00 [1.00, 1.00] (12,259/12,305) | 0.98 [0.98, 0.99] (2,183/2,220) | 0.99 |

Not ranked: Penthera 1.9.0 (the previous release, for comparison); credsweeper 1.19.4 (trained on this dataset, not comparable: made by the CredData authors, its ML model was trained with CredData).

## Where another tool does better than Penthera

- Precision: kingfisher 0.96, Penthera 0.89. Overall, kingfisher scored 0.45.
- Recall: deepsecrets 0.77, Penthera 0.64. Overall, deepsecrets scored 0.78.

## Sources

- Dataset: [Samsung CredData](https://github.com/Samsung/CredData)
- Numbers: https://github.com/danoszz/penthera/blob/main/bench/public/published.json (measured 2026-10-25; Sealed test half, scored once (E11b § 10).)
- Protocol: https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E11b-improvement-study.md
- All four benchmarks: https://penthera.org/benchmarks.md