# Penthera research

> Rules decide. Every study is registered before it runs, and every change is dated, so you can repeat it and challenge it.

## Featured: the public benchmark (2026-10-25)

Penthera and 25 free security tools, scored once on the sealed half of four public datasets ([all four](https://penthera.org/benchmarks.md)):

- Leaked secrets, Samsung CredData: 2nd of 10, F1 0.74
- Malicious npm packages, DataDog's dataset: 1st of 6, tied 1st to 3rd, caught minus false alarms 0.71
- Malicious PyPI packages, DataDog's dataset: 1st of 8, tied 1st to 3rd, caught minus false alarms 0.82
- Injection bugs in JavaScript libraries, SecBench.js: 1st of 8, tied 1st to 4th, share found in the right file 0.70

Where a range follows the rank, Penthera's rank fell in that range in 95% of bootstrap redraws.

## Every study, result and note, newest first

- 2026-10-29 · Study E20 · Registered: [E20: How safe is the web's front door?](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E20-front-door.md). How do the most visited sites, and recent startups, score on the live check's passive front-door checks? Aggregates only.
- 2026-10-29 · Study E21 · Draft: [E21: Would the rules have flagged it before the exploit?](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E21-rules-before-exploit.md). Would Penthera's free, offline rules flag known server-side bugs in open-source code before they were exploited? Vulnerable and fixed commits, scanned locally. No AI model is run.
- 2026-10-28 · Study E15 · Draft: [E15: prompt injection hidden in what coding agents read](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E15-hidden-prompt-injection.md). Can a static, rule-based check find instructions planted in skills, rule files and MCP tool descriptions, without flagging normal ones?
- 2026-10-28 · Study E17 · Result: [E17: the registered run on macOS and Ubuntu made no outside connection and gave the same result, for every command in scope](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E17-offline-and-determinism.md). URL scans are covered only with --offline: without it, a URL scan downloads the Retire.js database. git runs outside the guard. [Result](https://github.com/danoszz/penthera/blob/main/bench/offline/RESULTS.md#registered-run-28-oct-2026). Re-run 29 Oct with the exact-path normaliser: verdict unchanged. [Re-run](https://github.com/danoszz/penthera/blob/main/bench/offline/RESULTS.md#deviation-the-normaliser-29-oct-2026).
- 2026-10-28 · Post: [How we benchmarked Penthera against 25 free tools, losses included](https://penthera.org/blog/sealed-half-benchmark). We tuned on one half of four public datasets and scored once on the other, sealed half. What that method is, what it found, and how far to trust it.
- 2026-10-28 · Post: [We benchmarked 10 free secret scanners on 2,220 labelled secrets. We came 2nd.](https://penthera.org/blog/secret-scanners-creddata). gitleaks, trufflehog, deepsecrets, Nosey Parker and five more on Samsung CredData's sealed half, with intervals, the limits, and where Penthera loses.
- 2026-10-25 · Study E14 · Result: [E14: merging the free tools found more secrets but cost more than 0.02 precision, so --giants keeps Penthera-first](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E14-giants-combination.md). Does one report from Penthera and the free tools find more than Penthera alone, without losing more than a fixed margin of precision? The answer sets the default for --giants. [Result](https://github.com/danoszz/penthera/blob/main/bench/public/RESULTS.md#e14-step-2-the-sealed-test-half-scored-once-2026-10-28).
- 2026-10-24 · Study E11 · Result: [E11: Penthera 1.9.0 missed all five registered targets on the three public datasets](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E11-public-benchmarks.md). One public benchmark per capability, with answer keys we had no hand in: secrets on Samsung CredData, code sinks on SecBench.js, malicious packages on DataDog's dataset. [Result](https://github.com/danoszz/penthera/blob/main/bench/public/RESULTS.md#summary).
- 2026-10-24 · Study E11b · Result: [E11b: 2nd on secrets, tied 1st to 3rd on npm and PyPI malware, tied 1st to 4th on JS injection](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E11b-improvement-study.md). Tune on a dev half, then score a sealed test half once. Only the test half's numbers are published as the after. [Result](https://penthera.org/benchmarks).
- 2026-10-22 · Study E10 · Draft: [E10: Penthera on known-vulnerable apps](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E10-known-vulnerable-apps.md). What share of Penthera's findings are correct on deliberately vulnerable apps, and what share of their documented bugs does it find?
- 2026-10-22 · Case study: [eMudhra's broken CA keys show in the public key, and the TLS check now looks for them](https://github.com/danoszz/penthera/blob/main/docs/research/emudhra-weak-keys-2026-10.md). Each private key was an integer below 256, so a passive handshake is enough to find it. A match is proof, so the finding is confirmed.
- 2026-10-22 · Case study: [GitLab's critical patch: a passive scan cannot read a self-managed GitLab's version](https://github.com/danoszz/penthera/blob/main/docs/research/gitlab-version-exposure-2026-10.md). Asset fingerprinting is ruled out. The note proposes a rule that reads the version from the repository instead; no rule shipped.
- 2026-10-21 · Case study: [PHP's 21 Oct flaws: Penthera checks the version a site shows, and says when it cannot tell](https://github.com/danoszz/penthera/blob/main/docs/research/case-study-php-2026-10.md). Released in 1.8.0. A hidden version is never reported as safe, and the fixed versions match the ones in OpenAI's own checker.
- 2026-10-20 · Study E1 · Registered: [E1: Penthera's accuracy on real code](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E1-real-world-accuracy.md). How accurate is Penthera's static analysis on real web apps and agent setups its authors did not write? Precision and recall per category, each with a 95% interval.
- 2026-10-20 · Study E8 · Draft: [E8: Do harness guardrails reduce judge tampering?](https://github.com/danoszz/penthera/blob/main/docs/research/prereg/E8-harness-guardrails.md). When an AI coding agent gets benign tasks, some of which cannot be solved, do an AGENTS.md block and protected judges change how often it tampers with the checks?
- 2026-10-20 · Case study: [2026 agent incidents: six gaps for Penthera, from verifier integrity to agent sandboxes](https://github.com/danoszz/penthera/blob/main/docs/research/agent-incidents-2026.md). Agents under evaluation reached real systems they were never meant to touch. Each lesson ends in an action, and the draft study E8 tests one of them.

Statuses: Registered, Running, Result, Draft. A draft is not registered.