How a result earns its status here

Rule one

The specification is frozen before the result exists

Before a measurement runs, the thing it is measuring is written down and hashed: which data, which window, which yardstick, which costs, what would count as a pass, and what would count as a failure. Then it runs. The file is never edited afterwards.

The reason is unglamorous. A bar you can move after seeing the number stops being a bar. Nobody moves one on purpose. What people do is notice, entirely sincerely, that the original bar was poorly chosen, which is a thought that only ever arrives in one direction.

When a frozen specification turns out to have a defect in it, the defect is recorded and the specification is left alone. One experiment carried a rule whose bar got easier to clear as the thing being tested got better. Holding one quantity fixed, a candidate delivering 0.50 percentage points a year passed and one delivering 50 was rejected for being explained away. That specification still sits in the repository with its original hash and its original verdict, and the re-reading of it is logged as a separate entry.

Rule two

Every attempt is logged, including the ones that failed

Test twenty coins for fairness and one will look rigged. Same with portfolios, and the only defence is counting.

The ledger holds 134 entries across 47 runs and 23 experiment families. Nine runs reached a rejection, nine reached "we can't tell", 21 are exploratory, and 8 never reached a terminal verdict at all. Abandoned attempts stay in. So does the run that was thrown away because someone spotted a bug in it afterwards.

That count is then used against the results. A backtest quoted without its search count is not evidence. Keeping the count is the only way to have one.

The ledger is also the reason there is a rung for exploratory work. Under the protocol here, exploration establishes a mechanism, its rough size, its costs and its limits, and it explicitly cannot support a claim the site ships as settled. Evaluation freezes the specification first. Promotion needs independent or forward evidence on top of that. Twenty-one of the runs are exploratory, which is a large fraction of the corpus and a fair description of how much of this is finished.

Rule three

Costs go inside the rule, before any claim about what you would have got

A strategy's return is easy to compute and mostly fictional. What matters is what it pays after the fee, the spread, the tax on the trades it forces, and the trades themselves.

So a claim about what an investor would have received has to come out of a rule you could have executed, with the costs charged inside the loop rather than subtracted from the answer at the end. The two are not the same, because a cost charged inside the loop changes which trades the rule makes.

This cuts in both directions and the site reports it both ways. A cost assumption of 20% to 40% turnover a year was carried over to funds an investor would actually buy, implying 0.20 to 0.68 percentage points a year of trading cost. Reading the funds' own filings instead: eight of nine US systematic value funds file 5% to 9% turnover a year, which is 0.09 to 0.15 points and only 0.05 above the plain index fund. The assumption was right about how an index reconstitutes and wrong about the products on the shelf, and the correction made the case better rather than worse.

Rule four

A null is compared against what the design could have seen

"We found nothing" and "we could not have found anything" are different sentences, and only one of them is a finding.

Before a result showing no effect is interpreted, the design is asked how large an effect it could have resolved at all. The moving-average timing rule is the cleanest case on the site. It beat a properly matched control by 0.74 percentage points a year, and the smallest gap that design could reliably have separated from luck was 3.03. Both numbers are real. The verdict is that we cannot see, and publishing the 0.74 on its own would be reporting the width of our own ignorance as a result.

The rule is also applied to the project's own bars. One experiment rejected seven candidates against a bar of 0.30 percentage points a year on a design whose smallest visible effect was 1.039, printed on the same page. A verdict may not be stronger than the instrument that produced it, so six of ten verdicts were restated and the specification was recorded as defective rather than rewritten.

Rule five

When a number and its check disagree, both sides get investigated

Every piece of arithmetic here is pinned by a test against an independently computed value. Sometimes the two disagree. The tempting move at that moment is to widen the tolerance until the test passes, and it is forbidden here.

Instead the fixture, the implementation, the units, the tolerance and the conditioning are each examined on their own, the broken one is fixed, and the evidence is written down. Sometimes the answer is that neither is broken and the disagreement is the finding. Rebuilding the five-factor size series from its published parts left a residual of 3.517 percentage points a month against a tolerance of 0.015, which established that the five-factor size series is a different object from the three-factor one that shares its name. The failed check is the result.

The first thing this project ever ran was a reproduction gate against a published table. Thirteen of fifteen cells reproduce. Two do not, against two independently typeset editions of the same source, so the disagreement is systematic rather than luck. That gate is still recorded as unresolved, four months and 46 runs later, because pretending otherwise would have been the first correction on the list.

Machines

How AI tools are used here

Heavily. Saying so is more useful than the alternative, which you would work out anyway.

Almost every line of Python, TypeScript and prose in this repository was drafted by a large language model working under the written agreement linked at the top of this page. The analyses were run by machine. The tests were written by machine. These sentences were drafted by machine, then cut and rearranged by a person who is responsible for what they say.

What is checked, and how

The checking is deliberately mechanical, because a model that wrote a number is exactly the wrong reviewer for it.

• Every search over portfolio weights runs in Python under a frozen specification with a ledger entry. The site itself only ports closed-form arithmetic, and each port is tested against fixtures generated by the Python it came from.
• No number is typed into a paragraph. A sentence names a record, the record carries the value, its range, its date, its status and the file that owns it, and the build fails on a record that names a file or a heading which does not exist. Rename a section in the research and the site stops building.
• Data is hashed on the way in, and the hash goes in the ledger. That identifies the bytes used. It does not establish that the source was correct or that it was available at the time, and the difference has bitten this project.
• A prose linter fails the build on the vocabulary language models reach for, and a second check fails any performance figure written without naming what it was measured against.
• No page is published unread. Every one of these was read line by line by a person before it went up.

What none of that catches

A model produces a plausible wrong number far more readily than an implausible one, and plausible wrong numbers survive review by other models.

That review is the origin of four entries on the corrections log, including the largest one. It is also the reason the site's method now leans on things a model cannot talk its way past: a frozen hash, a build that fails, a test with a number in it that came from somewhere else.

Limits

What none of this protects you from

All of the above is process. Process is cheap insurance against a few specific failures and no insurance at all against the rest.

The data is US-heavy and stops where the files stop. The tax arithmetic assumes a US federal individual investor and ignores every state. Nothing here has been reviewed by a securities lawyer, a tax adviser or an accountant. No fund return series and no per-fund exposure vector is committed to the repository, because the licences do not allow it, so the site's own backtest runs on data a reader supplies.

And the largest limit is not a process failure. Most of the questions this site asks cannot be settled by the data that exists, and no amount of care about specifications changes that. How sure we are puts numbers on it.