How we run detection tests.

What a run is

A run is one dated session in which a set of browsers visits a set of detectors through a set of proxy exits, and the outcome of every visit is written down, including the visits that failed and the ones our own tooling broke. Runs are published monthly. Each one is a sample of a moving target, not a certification: detectors change, addresses change reputation, and a result from last month is evidence about last month.

Everything a run touches is public and none of it requires anyone's cooperation, so the whole thing is repeatable by a reader with a free account and an afternoon.

Environment

• Profiles are fresh unless the run says otherwise. Created, used once, discarded. That is deliberately the weakest realistic case: real accounts carry cookies, history and behaviour a first-visit profile does not have.
• Nothing is tuned per target. Default settings. A result produced by hand-tuning for one site says nothing about the product anyone else installs.
• Headful, on a real desktop OS. The host OS is recorded in the dataset, because it is a variable: some tells are properties of the machine underneath rather than of the browser on top.
• The engine build is recorded per check. New profiles bind to the newest published build rather than a frozen one, so two segments of the same run can differ, and pretending they did not would hide a variable.

Subjects

A single-product screenshot is not a measurement. Every run that compares anything states what it compared against, and the baseline is chosen to be the thing readers actually weigh us against rather than the thing we look best next to. In practice that means an open-source automation stack with a stealth plugin: free, widely used, and specifically built to hide automation tells.

The baseline is left unhelped on purpose. No timezone override, no locale override, no proxy wiring borrowed from us. Handing it our configuration would measure our configuration twice. Where the baseline leaks something that could be fixed by hand, the run says so: the question is who keeps that list current across browser releases, not whether a fix exists.

Result vocabulary

Five values, and they are not interchangeable. Collapsing them into pass/fail is what turns a measurement into marketing.

Two consequences worth stating. A check our harness broke is inconclusive or error, never quietly dropped and never rounded up to a pass. And a detector that let a browser through while that browser announced a contradiction is recorded as both: the detector passed it, the coherence check failed it.

Proxies, and why some detectors need a different design

The exit address is a variable, and on some checks it is the dominant one. Anything that scores address reputation cannot be measured by sending two browsers through the same address minutes apart: the second visitor from an address that was just used on the same target is in a different situation from the first, and a single burned address blocks both arms and looks like agreement.

So a stateful detector gets a paired design:

• every visit gets its own exit address, never reused for any purpose;
• the two subjects in a round draw different addresses from the same /16, holding the neighbourhood constant instead of leaving it to chance;
• the arm that goes first alternates by round;
• every visit uses a fresh profile, since one previous visit leaves cookies that make the next one a returning visitor.

Stateless checks do not need this and are run through a single shared exit, which is why one run can carry segments with different designs. The dataset keeps them apart; averaging across them would produce a number nobody measured.

Timezone, locale and geolocation follow the exit address automatically in our product. Where a subject's timezone or language disagrees with its exit, that is a recorded result, not a configuration mistake we corrected before publishing.

Known biases

• Small n. A handful of rounds against one target is not a rate. Where a run reports something like five out of six, it says in the same breath that one split is not evidence of anything.
• Homepages, not checkouts. Commercial walls score login, search and checkout endpoints far more harshly than a landing page. Unless a run says otherwise, it visited the front door.
• One host, one operator. Every published run so far comes from one machine on one network. If your result differs from ours, distrust ours first.
• We are not a neutral party. We publish the address next to the score, the failures next to the passes and the raw data next to the prose precisely because the incentive runs the other way.

Corrections

Retractions stay on the page and in the data. When a later measurement contradicts something an earlier run asserted, the original paragraph is left standing and the correction is written next to it, and the machine-readable file carries the same retraction in its notes. A report that quietly edits away the parts that turned out wrong is worth less than one that keeps them.

The data files

Every run publishes the same three things: the prose, a JSON file, and a CSV of the same rows. Field names are fixed once published; a correction ships as a new file rather than a silent rewrite.

All of them are served with Access-Control-Allow-Origin: *, so they can be fetched from your page rather than retyped into it. The field-by-field reference lives on the dataset page.

Licence

The results and this method page are published under CC BY 4.0. Use them anywhere, including commercially, including in a comparison that disagrees with us. The one condition is attribution, and every run page carries a ready-to-paste citation at the bottom.

Most recent run: Detection run, August 2026.