Essay · August 2026

I diffed my product against its own terms of service and found fifteen contradictions

One day in August I stopped reading my code and started reading my terms of service. Not as a contract, but as a specification: every sentence in it that describes behaviour is a claim about the software. Fifteen of them did not hold.

Day 7

On day 7 the dashboard required proof of work. My own terms describe the check-in on that day as voluntary.

So anyone who had read the contract and believed it would have uploaded nothing — and walked straight into the consequence designed for people who do nothing at all. The product would have punished someone for trusting a commitment I made.

Nobody was affected. As of August 2026 no paying customer has ever started a sprint in this product, and there is no testimonial. That is not an excuse — it is the reason any of this can be written down. All fifteen findings hit nobody, because nobody is in there yet. There is no better moment coming.

Why I looked

This was not a code review. I suspected my external communication and my product had drifted apart, and I wanted to know by how much. So I took the only text that is legally binding and treated it as a requirements list.

Four steps, short enough to copy:

1. Read the terms and the privacy policy as a specification, not as legalese. Mark every sentence that makes a statement about behaviour — “X is recommended but not mandatory”, “a suspension occurs when …”, “we typically reply within …”.

2. For each marked sentence, find the place in the product that executes it. Not the place that mentions it — the place that enforces it.

3. Search for those same sentences across everything customer-facing: landing page, FAQ, checkout, emails, the machine-readable surfaces. A sentence rarely exists once.

4. For every finding, record the direction: does the code deviate in the user’s favour or against them?

Step 4 is the one I almost skipped, and it is the most important. It separates a legal problem from a fairness problem — and it explains why half of the findings survived as long as they did.

Three classes, not a bug list

Class 1 — the contract says A, the code does B

Day 7 belongs here. So does a second case: the product suspended a participant after two skipped days, and my contract did not provide for that — it enumerates the grounds for suspension exhaustively, and this one was not among them.

That is explicitly not a liability finding, and the entire difference is the direction: the code was stricter than the contract. The dangerous case is the mirror image — a contract promising something the software does not deliver. Same size of deviation, two completely different consequences.

Class 2 — the marketing says A, the contract says B

The leading example: in four places I advertised a response time that was shorter than the one I had contractually committed to. Nobody complains about a promise that is too generous. The error is silent by construction — there is no complainant for an oversized promise, right up until the one day you fail to keep it.

I cannot quote that sentence here. It now sits on a blocklist that runs across the repository on every build — and this file is not exempt from it. I tried while writing, and the check went red. That is not a mishap, it is the intended behaviour, and it is the most honest evidence I can offer for the rest of this text.

The same class holds the sentence I liked least. In six places it said the participant receives a personal email. What went out was a template. The sentence was not a lie when it was written — it became one the moment I automated the sending. There were exactly two honest resolutions: write the email by hand, or stop promising it. Today the sentence appears in none of those six places.

Class 3 — statements nothing can structurally contradict

Two code paths could never fire. Not broken — unreachable. No test went red, because no test exists that asserts “this line is ever reached”. No error was raised, because a path that never runs never fails.

This is the expensive class. It is indistinguishable from working software using every tool I have. I found neither by reading; I found them by running the whole chain end to end, once.

Fifteen

I found eleven instances by hand that working day. Three more turned up while proofreading the list — that is, inside the document I had created in order to close the matter. At fourteen I considered it complete.

That same evening I built the check mentioned earlier and ran it for the first time. It immediately reported a fifteenth occurrence — same class, same wording, in a file neither I nor the proofreading pass had looked at.

Fifteen is not a decay rate. It is what a first measurement looks like. Fifteen instances of one class, in a product with no paying sprint behind it, all corrected the same day and verifiable. And the last of them was found not by me but by the first tool capable of looking for it. If you run this yourself and come out at zero, you probably did not measure — you confirmed.

The pattern

A comment has no assertion. A README has no CI. Your terms of service are a test suite nobody runs.

That is the actual discovery, and it is not that fifteen statements were wrong. It is that in a product like this two specifications exist and only one of them is tested — and the untestable one is the one that binds. The code has tests, linters, types, a pipeline and a reviewer. The text that governs in a dispute has none of it. It has an author and a date.

So the class does not come from carelessness. It comes from the same rule existing in four, six, eleven places, and none of them stating how many there are. Every change is an n-place operation with an unknown n. Someone changing a number from 24 to 48 is not asking a positioning question — so they do not ask anyone.

Two findings, then onward

Kleven et al. (2011) ran a randomised audit experiment with more than 40,000 Danish taxpayers. Evasion was close to zero on third-party-reported income and substantial on self-reported income. Same people, same incentives; the only difference is whether somebody else reports the figure. Scheel, Schijen and Lakens (2021) compared 152 standard psychology studies with 71 Registered Reports: 96 per cent positive results against 44. Once the hypothesis is fixed before the result — once it can fail — the success rate halves.

Both describe the same property, once in money and once in science: a statement that nothing can contradict stops behaving like a statement. Documentation is self-reported by default. That is not a question of character — it is a property of the reporting arrangement.

What this has to do with the product

In the sprint I sell, the goal is written down verbatim before the start and can afterwards only be corrected downwards. At each milestone a human reads what was delivered.

For a long time I believed that works because a human reads. After that day I believe it works because the reading happens against something that was fixed beforehand. A reviewer without a frozen yardstick reviews nothing. In those fifteen cases no frozen yardstick existed — there was simply nothing to check against.

So that this does not pass as advertising: having a person read the proof is a scarcity, not a defensible advantage. It costs someone’s time, and time does not duplicate the way software does. I ran the sprint myself as a participant, with a definition of “done” fixed in advance — and missed one of four criteria. That is public, because a flawless self-experiment would just be another claim.

What I have had since 8 August is the check from the section above: a small number of hard rules that fail the build when a retired sentence reappears. Deliberately small — a gate that mostly reports noise gets ignored within two weeks. It checks wording, not correctness. It knows strings, not meaning, so it cannot tell a claim from its negation. And absence is invisible to it: a promise that quietly disappears produces no match. Of the fifteen cases it would have caught roughly a third — the cheap ones. One of them it actually did catch, on its very first run; that was the fifteenth. It would have let the four most expensive ones through.

The twenty-minute version

Take the longest document that describes your product and is not code: terms, README, landing page, onboarding email. Mark every sentence that makes a statement about how your software behaves. Take three of them and find the place that executes them. Record the direction for each.

And the condition without which the rest is worthless: write down beforehand which claim you are checking and what result would refute your assumption. Otherwise you will end up confirming what you expected — that is the distance between 96 and 44 per cent, and it does not come from dishonesty.

Which fixes my next check, and it can go wrong: in a few weeks I will look at whether the fifteen corrected sentences still hold. If more than two of them have drifted again, then the build check was the wrong answer — the problem would be the number of places rather than the supervision of them, and the correct response would be to delete carriers instead of monitoring them.

If reading your own sentences back raised the question of who actually notices when you skip day 4: five clicks will tell you which module would fit — and whether any of them does.

Take the quiz →

FAQ

What does it mean to read your terms of service as a test suite?

It means treating every sentence in the contract that describes behaviour as a claim about the software, and then looking for the place in the product that executes it. A contract clause such as "daily check-ins are recommended but not mandatory" is functionally an assertion: it says the system must not punish someone for skipping one. Nothing in a normal build ever evaluates that assertion. The terms are a specification with no test runner attached, which is why a product can drift away from them for months without a single error appearing anywhere.

What were the three classes of contradiction?

First, contract says A and code does B — the contract described something as voluntary while the product required it. Second, marketing says A and the contract says B — an advertised commitment that was stricter than the contractually agreed one, and a promise about a human action that was in fact performed by a template. Third, statements that nothing could structurally contradict — two code paths that could never be reached, so no test could ever go red and no error could ever be raised. The third class is the expensive one, because it looks identical to working software.

Were customers affected?

No, and that is the only reason this can be written down at all. As of August 2026 no paying customer has ever started a sprint in this product and there is no testimonial. Fifteen instances of one class, in a product with no paying sprint behind it, all corrected on the same day. That is not a decay rate — it is what a first measurement looks like, taken at the only point where a first measurement is free.

Why is the direction of a deviation more important than the deviation itself?

Because it separates a legal problem from a fairness problem. When the code is stricter than the contract, the drift works against the provider: nobody complains about a promise that is too generous, so the error is silent by construction and survives for as long as nobody audits it. When the code is more permissive than the contract, the same size of deviation is a liability. The measurement is the same; the consequence is not. Recording the direction for every finding is what turns a defect list into an audit.

What does research on third-party reporting have to do with documentation?

Kleven et al. (2011) ran a randomised audit experiment with more than 40,000 Danish taxpayers and found evasion close to zero on third-party-reported income and substantial on self-reported income — same people, same incentives, the only difference being whether somebody else reported the figure. Scheel, Schijen and Lakens (2021) compared 152 standard psychology studies with 71 Registered Reports and found 96 per cent positive results against 44 per cent. Both describe the same property: a statement that cannot be contradicted by anything stops behaving like a statement. Documentation is self-reported by default.

What can an automated check for this actually catch?

The cheap, repeating cases and nothing else. A string check knows characters, not meaning: it cannot tell a claim from its negation, it cannot judge whether a sentence is true, and absence is completely invisible to it — a promise that quietly disappeared produces no match at all. Of the fifteen findings it would have caught roughly a third, all of them from the class that costs a reader rather than a customer. The expensive ones needed a dry run of the whole chain, and a dry run is manual work.

What is MVP Builder?

A structured 13, 21 or 30-day sprint for developers with a full-time job whose side project is stuck. Every evening a prompt for the next day is generated from the project context and the previous check-in, and at each milestone a human reads what was actually delivered before the sprint continues. The entry point is a five-question click quiz at mvpbuilder.io/go, followed by a free 30-minute intro call.

Sources

Kleven, Knudsen, Kreiner, Pedersen & Saez, Unwilling or Unable to Cheat? Evidence From a Tax Audit Experiment in Denmark, Econometrica 79(3), 2011, pp. 651–692 — doi.org/10.3982/ECTA9113
Scheel, Schijen & Lakens, An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports, Advances in Methods and Practices in Psychological Science 4(2), 2021 — doi.org/10.1177/25152459211007467
The contract clauses checked against are public: mvpbuilder.io/legal (terms section 4.1 on daily check-ins, section 5 on grounds for suspension, section 12 on the response deadline).