Research

Rules decide. Every study is registered before it runs, and every change is dated, so you can repeat it and challenge it.

10 studies · 4 case studies · 2 posts. Statuses: Registered, Running, Result, Draft. A draft is not registered.

FeaturedPublic benchmark

Penthera and 25 free security tools, on four public benchmarks

Four public datasets other people built, each split in half before any tuning, and scored once on the sealed half. Penthera's rank on each:

  1. Leaked secrets Samsung CredData

    2nd of 10

    F1 0.74

  2. Malicious npm packages DataDog's dataset

    1st of 6, tied 1st to 3rd

    caught minus false alarms 0.71

  3. Malicious PyPI packages DataDog's dataset

    1st of 8, tied 1st to 3rd

    caught minus false alarms 0.82

  4. Injection bugs in JavaScript libraries SecBench.js

    1st of 8, tied 1st to 4th

    share found in the right file 0.70

Where a range follows the rank, Penthera's rank fell in that range in 95% of bootstrap redraws.

Every tool, interval and limit

Every study, result and note, newest first

  1. Study · E20Registered

    E20: How safe is the web's front door?

    How do the most visited sites, and recent startups, score on the live check's passive front-door checks? Aggregates only.

  2. Study · E21Draft

    E21: Would the rules have flagged it before the exploit?

    Would Penthera's free, offline rules flag known server-side bugs in open-source code before they were exploited? Vulnerable and fixed commits, scanned locally. No AI model is run.

  3. Study · E15Draft

    E15: prompt injection hidden in what coding agents read

    Can a static, rule-based check find instructions planted in skills, rule files and MCP tool descriptions, without flagging normal ones?

  4. Study · E17Result

    E17: the registered run on macOS and Ubuntu made no outside connection and gave the same result, for every command in scope

    E17: Does Penthera work unplugged, and give the same answer twice?

    URL scans are covered only with --offline: without it, a URL scan downloads the Retire.js database. git runs outside the guard. Read the result

    Re-run 29 Oct with the exact-path normaliser: verdict unchanged. Read the re-run

  5. Post

    How we benchmarked Penthera against 25 free tools, losses included

    We tuned on one half of four public datasets and scored once on the other, sealed half. What that method is, what it found, and how far to trust it.

  6. Post

    We benchmarked 10 free secret scanners on 2,220 labelled secrets. We came 2nd.

    gitleaks, trufflehog, deepsecrets, Nosey Parker and five more on Samsung CredData's sealed half, with intervals, the limits, and where Penthera loses.

  7. Study · E14Result

    E14: merging the free tools found more secrets but cost more than 0.02 precision, so --giants keeps Penthera-first

    E14: Penthera plus the free tools, against each alone

    Does one report from Penthera and the free tools find more than Penthera alone, without losing more than a fixed margin of precision? The answer sets the default for --giants. Read the result

  8. Study · E11Result

    E11: Penthera 1.9.0 missed all five registered targets on the three public datasets

    E11: Penthera on public benchmarks

    One public benchmark per capability, with answer keys we had no hand in: secrets on Samsung CredData, code sinks on SecBench.js, malicious packages on DataDog's dataset. Read the result

  9. Study · E11bResult

    E11b: 2nd on secrets, tied 1st to 3rd on npm and PyPI malware, tied 1st to 4th on JS injection

    E11b: improving Penthera on E11, without fooling ourselves

    Tune on a dev half, then score a sealed test half once. Only the test half's numbers are published as the after. Read the result

  10. Study · E10Draft

    E10: Penthera on known-vulnerable apps

    What share of Penthera's findings are correct on deliberately vulnerable apps, and what share of their documented bugs does it find?

  11. Case study

    eMudhra's broken CA keys show in the public key, and the TLS check now looks for them

    Each private key was an integer below 256, so a passive handshake is enough to find it. A match is proof, so the finding is confirmed.

  12. Case study

    GitLab's critical patch: a passive scan cannot read a self-managed GitLab's version

    Asset fingerprinting is ruled out. The note proposes a rule that reads the version from the repository instead; no rule shipped.

  13. Case study

    PHP's 21 Oct flaws: Penthera checks the version a site shows, and says when it cannot tell

    Released in 1.8.0. A hidden version is never reported as safe, and the fixed versions match the ones in OpenAI's own checker.

  14. Study · E1Registered

    E1: Penthera's accuracy on real code

    How accurate is Penthera's static analysis on real web apps and agent setups its authors did not write? Precision and recall per category, each with a 95% interval.

  15. Study · E8Draft

    E8: Do harness guardrails reduce judge tampering?

    When an AI coding agent gets benign tasks, some of which cannot be solved, do an AGENTS.md block and protected judges change how often it tampers with the checks?

  16. Case study

    2026 agent incidents: six gaps for Penthera, from verifier integrity to agent sandboxes

    Agents under evaluation reached real systems they were never meant to touch. Each lesson ends in an action, and the draft study E8 tests one of them.

A study is registered by the commit that merges its protocol, and the protocol lists any data seen before that. Changes after that are logged in its deviations section, with the date and the reason. Everything here is read from docs/research, bench and the blog when the site is built.