We benchmarked 10 free secret scanners on 2,220 labelled secrets. We came 2nd.
Which free secret scanner finds the most leaked secrets, on data none of them wrote?
On the sealed test half of Samsung CredData (2,220 labelled secrets), deepsecrets ranked first of 10 free tools with F1 0.78. Penthera 2.0.0 ranked 2nd with 0.74. Nosey Parker was close behind with 0.69. Read the limits below before you use any of these numbers. Every tool, with its counts, is on the secrets benchmark page.
The ranking
F1 balances precision (how many reported secrets were real) and recall (how many real secrets were reported). Intervals are 95% Wilson.
- deepsecrets: F1 0.78. Precision 0.80 (0.79 to 0.82), recall 0.77 (0.75 to 0.78).
- Penthera: F1 0.74. Precision 0.89 (0.87 to 0.90), recall 0.64 (0.62 to 0.66).
- Nosey Parker: F1 0.69. Precision 0.76, recall 0.62.
- gitleaks: F1 0.52. Precision 0.85, recall 0.37.
- ripsecrets 0.45, kingfisher 0.45, detect-secrets 0.38, secretlint 0.32, whispers 0.15, trufflehog 0.10.
The median F1 of the other free tools was 0.45.
Where Penthera loses
deepsecrets finds more. Its recall is 0.77 against Penthera's 0.64, and the intervals do not overlap. Penthera reports fewer false hits: precision 0.89 against 0.80. If missing a secret costs you more than reading a false alarm, that trade goes the wrong way for you.
kingfisher has the highest precision of the ranked tools, 0.96, at a recall of 0.29.
gitleaks and trufflehog
These two are what most teams run, and the comparison pages people search for have no measurements. Here, gitleaks scored F1 0.52 and trufflehog 0.10. That gap undersells trufflehog, for two reasons:
- trufflehog's main feature is verifying a key with its provider. Every tool ran offline, so verification was off.
- CredData replaces every real secret with random characters of the same shape. A verified-secrets tool has nothing real to verify, and a pattern tool loses nothing.
So read this as "gitleaks finds more look-alike secrets in source files than trufflehog does without verification", not as "gitleaks is the better tool".
Not ranked
CredSweeper scored F1 0.99. It is made by the CredData authors, and its model was trained with CredData, so it is shown and not ranked. Semgrep's secrets rules were skipped: they download from the Semgrep registry at scan time and cannot run offline.
The limits
- CredData's labels lean partly on other scanners' output.
- CredData is split by file, so most repositories have files on both halves. The leakage audit found that removing the one repository it flagged moves Penthera's F1 from 0.742 to 0.738.
- Penthera was tuned on the dev half; the other tools ran on their defaults.
- Paid scanners (GitGuardian, Socket, Snyk, Phylum, Semgrep Pro) publish no results on CredData, and we cannot measure them.
- CredData is not your code.
Check it yourself
- Every number: published.json and the secrets page.
- The method and the protocol: how we benchmarked against the free tools, and E11b.
- Scan your own repo for secrets:
npx penthera@2.0.0 --repo .
Sources: published.json, RESULTS.md, E11b-improvement-study.md.
penthera.org · Benchmarks · Blog · Live check · Terms · Trust · Source · MIT license