Free secret scanners on Samsung CredData: 10 tools ranked
On the sealed test half of Samsung CredData (2,220 labelled secrets), deepsecrets ranked 1st of 10 free tools, with F1 0.78. Penthera 2.0.0 ranked 2nd with 0.74: better precision (0.89 against 0.80), worse recall (0.64 against 0.77). Measured on 25 October 2026.
Which free secret scanner finds the most leaked secrets on CredData?
Penthera 2.0.0 ranked 2nd of 10 free tools by F1 (0.74), behind deepsecrets (0.78). Scored by F1 of strict precision and in-scope recall, as the CLI reports.
- deepsecrets0.78
- Penthera0.74
- noseyparker0.69
- gitleaks0.52
- ripsecrets0.45
- kingfisher0.45
- detect-secrets0.38
- secretlint0.32
- whispers0.15
- trufflehog0.10
Every tool, with 95% intervals
Counts are found over total. Intervals are 95% Wilson intervals. Penthera's row is bold.
| Rank | Tool | Precision | Recall | Score: F1 |
|---|---|---|---|---|
| 1 | deepsecrets 2.2.0 | 0.80 [0.79, 0.82] 2,752/3,427 | 0.77 [0.75, 0.78] 1,704/2,220 | 0.78 |
| 2 | Penthera 2.0.0 | 0.89 [0.87, 0.90] 1,640/1,849 | 0.64 [0.62, 0.66] 1,415/2,220 | 0.74 |
| 3 | noseyparker 0.24.0 | 0.76 [0.74, 0.78] 1,898/2,501 | 0.62 [0.60, 0.64] 1,387/2,220 | 0.69 |
| 4 | gitleaks 8.30.1 | 0.85 [0.84, 0.86] 2,242/2,635 | 0.37 [0.35, 0.39] 831/2,220 | 0.52 |
| 5 | ripsecrets 0.1.11 | 0.65 [0.64, 0.67] 1,883/2,877 | 0.35 [0.33, 0.37] 766/2,220 | 0.45 |
| 6 | kingfisher 2.14.0 | 0.96 [0.94, 0.97] 797/834 | 0.29 [0.28, 0.31] 653/2,220 | 0.45 |
| 7 | detect-secrets 1.5.0 | 0.30 [0.29, 0.31] 2,237/7,444 | 0.53 [0.51, 0.56] 1,187/2,220 | 0.38 |
| 8 | secretlint 13.0.7 | 0.92 [0.89, 0.94] 459/499 | 0.19 [0.18, 0.21] 424/2,220 | 0.32 |
| 9 | whispers 2.4.0 | 0.46 [0.43, 0.50] 281/605 | 0.09 [0.08, 0.10] 193/2,220 | 0.15 |
| 10 | trufflehog 3.99.3 | 0.40 [0.35, 0.45] 152/383 | 0.05 [0.05, 0.07] 122/2,220 | 0.10 |
| Not ranked | Penthera 1.9.0 | 0.56 [0.51, 0.60] 237/425 | 0.10 [0.09, 0.12] 233/2,220 | 0.18 |
| Not ranked | credsweeper 1.19.4 | 1.00 [1.00, 1.00] 12,259/12,305 | 0.98 [0.98, 0.99] 2,183/2,220 | 0.99 |
- A rank needs at least five other free tools measured; this benchmark has 9.
- Median score of the other free tools: 0.45.
Not ranked, and why
- Penthera 1.9.0: the previous release, for comparison. Score 0.18.
- credsweeper 1.19.4: trained on this dataset, not comparable: made by the CredData authors, its ML model was trained with CredData. Score 0.99.
Where another tool does better than Penthera
- Precision: kingfisher 0.96, Penthera 0.89. Overall, kingfisher scored 0.45.
- Recall: deepsecrets 0.77, Penthera 0.64. Overall, deepsecrets scored 0.78.
Tools that are not here
Paid tools (GitGuardian, Socket, Snyk, Phylum, Semgrep Pro) are not here. None publishes results on these benchmarks; we cannot measure them (cost and licence terms). Skipped: semgrep p/secrets (downloads its rules from the Semgrep registry at scan time; cannot run offline).
Published elsewhere, on other data
Not comparable with the numbers above: other datasets, other scoring.
Check it yourself
The dataset is Samsung CredData. The ranking rules were written down before the run: the protocol. Every number on this page is read from published.json when the site is built; the results file has the method and per-family detail, and the runner's README says how to reproduce it.
Measured on . Sealed test half, scored once (E11b § 10). All four benchmarks.
penthera.org · Benchmarks · Blog · Live check · Terms · Trust · Source · MIT license