Rule by Default

Every Saturday I ask 92 of the largest retail storefronts in America the same six questions. The sample is the NRF Top 100, minus the eight with no single storefront.

The questions are plain requests. Can I read your robots.txt? Do you publish an llms.txt, a commerce profile, an agent card? Will you serve me your homepage if I am not a browser? The scanner names Major Labs in every request and links to a page that explains itself. It waits 1.5 seconds between requests. If the door says no, it leaves and does not try again.

Here is what came back on September 26.

Twenty-eight of the 92 did not serve robots.txt. Twenty-two refused with a 403. Four never answered. One answered with a 418, the joke status code that means "I'm a teapot," and one served something that was not a robots file.

Of the 64 that did serve the file, 16 name any AI crawler or agent at all. Two block the kind of agent that acts for a live person.

Forty-nine served a homepage an agent can read. Sixteen publish an llms.txt. Three publish a Universal Commerce Protocol profile. None has an A2A agent card.

One limit belongs here and not in a footnote. My scanner is an unknown research tool that identifies itself. It is not a shopping agent from a company the retailer has heard of. What I measure is how a stranger who is honest about being software gets treated at the door.

The web has exactly one door policy that is written down in a standard place. It is robots.txt, and the people who wrote the standard were clear about what it is not.

"These rules are not a form of access authorization."

That is RFC 9309, published in 2022. Martijn Koster said the same thing in the original 1994 note: "It is not enforced by anybody."

So there are two layers. The rulebook says what a site wants. Something else does the refusing. My data lets me hold one against the other.

Of the 64 storefronts with a readable robots.txt, 62 have written no rule that blocks a user-triggered agent. Eight of those 62 still refused or challenged my scanner at the homepage. Five more handed back an empty shell. One returned a page too large to judge.

Two storefronts publish an llms.txt, a guide written for AI systems, and turned the scanner away at the homepage regardless.

The rulebook and the bouncer are not reading from the same page.

Here is a detail I did not expect. Under the standard, when a request for robots.txt gets a status code in the 400 range, the file counts as unavailable, and "the crawler MAY access any resources on the server." Twenty-two storefronts answered 403. Read by the book, a refusal to show the rules tells a standards-following crawler that there are no rules. I doubt any of the 22 meant that. The standard is written for crawlers, and "may" is not "should," so I would not lean on it in court. It shows how far the enforcement layer has drifted from the written one.

Week to week, the written layer did not move at all. All 92 storefronts gave identical answers on every written measure: who they name, who they block, which files they publish. The small changes I saw were all at the edge.

Now the weakness, stated as plainly as I can. My scan sees a refusal. It cannot see why. These are some of the largest retailers in the country. They have security teams and enterprise contracts, and their bot rules may be tuned by hand. If so, the refusals are decisions.

Then they are decisions nobody wrote down. Sixteen storefronts name an AI agent in writing, and two block one that acts for a person. Far more than two are turning it away.

What I can say today is that the pattern is consistent with a preset. I cannot say it is set by one. The scanner does not yet record which vendor answered the door. It will, and if the vendor predicts the answer better than anything about the retailer, that is evidence for the default. If it does not, this chapter gets weaker and I will say so.

Put the two measurements side by side.

At the storefront, an agent that says who it is gets refused before it can read the rules. At the tool server, a caller who says nothing can often write a file.

One side is shut and the other is open. The cause looks the same. In both places the outcome is consistent with a setting that was inherited, and in neither place did the party bound by it write anything down.

Each default is rational for whoever set it. A bot manager's preset was tuned years ago for scrapers and card testers, and for that job, refusing unknown software is correct. A protocol's first-run path was tuned for adoption, and for that job, asking nothing is correct. Neither was designed with a customer's agent in mind, because when they were designed there was no such customer.

So the person who sends an assistant to buy a coffee maker meets a locked door at the shop, and the anonymous caller at the workshop finds the tools laid out. I would not say the locks are on the wrong doors. I would say nobody decided where the locks go.

There is a long record on what people do with a default. Mostly they leave it.

The famous case is organ donation. In 2003, Eric Johnson and Daniel Goldstein compared countries where you must opt in to be a donor with countries where you must opt out. On effective consent rates, "the two distributions have no overlap, and nearly 60 percentage points separate the two groups." Same question. Different starting position.

Money behaves the same way. Brigitte Madrian and Dennis Shea studied a large US company that began enrolling employees in its retirement plan automatically. Participation was significantly higher, and a substantial fraction of those enrolled stayed at the contribution rate and the fund the company had picked for them.

A federal court has looked at the commercial value of the position. In United States v. Google, the court found that Google paid $26.3 billion in 2021 in revenue share under its distribution contracts, and it wrote a sentence that belongs on a wall.

"The default is extremely valuable real estate."

All of that is about people. My data is about code and configuration, and it says the same thing.

A week apart, 89 of 92 storefronts gave the same answer on robots.txt. I count readable, refused and no answer as three different answers, which is the strict way to count. On the looser test of readable or not, it is 91.

On the server side, I scanned 1,934 servers in both June and September. Fourteen weeks later, 1,668 of them had exactly the same number of findings. That is 86 percent. In that same group, 12 servers put authentication in front of a sensitive action. Six took it away. And 430 were unauthenticated in June and unauthenticated in September.

Then there is the part I did on purpose. In June I flagged 47 high-risk servers for disclosure and wrote to the maintainers of nine. One fixed the problem. Across the whole 47, as of September 27, two are fixed, four have fewer findings, 34 are unchanged and seven are worse.

The numbers are small, and I will not claim that writing to people makes no difference. I will re-score all 47 on December 12 and publish what I find. What I will say is that a polite letter did not move the line, and I no longer expect a thousand polite letters to.

If presets stay put, how does anything change? Look at the times it did.

In 2021, Apple made tracking on the iPhone something an app had to ask for. A working paper presented at the FTC measured what followed in the United States: the share of trackable Apple traffic fell "from 73% to 18%." The authors estimate a 21 percent fall in ad revenue from Apple users for publishers. Meta's finance chief told investors the iOS changes were a headwind "on the order of $10 billion" for 2022. That was the company's own estimate.

In April 2023, Amazon Web Services changed what a new storage bucket looks like. Public access is blocked unless the owner turns it on. Years of guidance had asked customers to lock their buckets. The default did it for every new one.

Defaults can open doors as well. In November 2025, AWS said its web firewall would let verified agents through: "Verified WBA bots will now be automatically allowed by default." In March 2026, Shopify told merchants, in an email reported by Modern Retail, that its agentic storefront channel "will launch by default for your store."

And sometimes the holder of a default announces a change and then does not make it. Google spent years preparing to change how Chrome handles third-party cookies. In April 2025 it said it had "made the decision to maintain our current approach."

I do not have a theory of why one of these stalled and the others went through. What I take from them is narrower. When the number moved, the party that held the default moved it, and it moved for everyone under that default at once.

Change arrives in rows.

If the setting is the rule, it is worth knowing who holds the settings. I count five kinds of holder, and I am not ranking them.

Edge and bot-management vendors. They sit in front of the website and answer the door. Their presets decide what an unknown visitor gets before the site's own code runs.

Protocol and SDK maintainers. They decide what a server does when the developer specifies nothing. "Optional" is a default.

Authors of starters. The quickstart, the README example, the template. Whatever is in the first 15 lines is in a great many servers, though I cannot yet tell you how many. I have no measurement of how many servers begin from a starter, and I am not going to guess.

Commerce platforms. They can switch a channel on for every store they host, as Shopify did.

AI companies. They choose how their agents announce themselves: a named user agent, a signed request, or a browser that looks like a person. That choice is a default too, and it shapes what every door on the street can do about them.

Then there is a sixth party, which holds nothing. The merchant who inherits the bot rule. The developer who inherits the starter. They are bound by the setting, and mostly they have not read it.

I am not arguing that these settings are wrong. A preset that refuses unknown software protects merchants from real abuse. An open first step is how protocols get used at all. My complaint is about legibility.

A good default has four properties.

It can be read. The bound party can find out what the setting is without filing a ticket.

It is published where it takes effect. If the door refuses agents, the rule that says so is at the door, in a form an agent can fetch.

It can tell who is asking. A signed request from a customer's assistant and an anonymous scraper should not look identical. The work on this is real but unfinished. AWS notes that Web Bot Auth "relies on two active IETF drafts."

It can be reversed. The opt-out exists and an ordinary owner can find it.

On the server side the ask is older and has a name. In 2023 the US cybersecurity agency CISA and its partners published guidance that says it in one sentence: "A secure configuration should be the default baseline." Applied here, that means the path that puts a server on a network includes the lock. The lock already exists in both SDKs. It needs to be in the first 15 lines that reach a network.

Here are the five best arguments against this booklet.

The scan cannot tell a preset from a decision. This is the strongest one, and chapter 3 concedes it. The index also did not move when Cloudflare's date passed. Two runs on September 26 gave 29 and 28, which is noise. My headline example and my headline number are not causally linked, and I have not said they are.

Defaults slip when someone powerful wants them to. Lauren Willis showed this in "When Nudges Fail." Retailers have a revenue reason to open the door. I agree that the doors will move. My claim is about how: in steps, when a holder acts.

Law is already here. It is, in part. Chapter 7 says where.

The default may be right. It may. I have argued for legibility and not for a particular setting.

The server figure overstates. The all-server count includes local servers behaving exactly as the standard intends. That is why I lead with the network-facing 288 of 384.

A theory that cannot be wrong is not worth much, so here is how to mark mine. Each line can be scored from my own series or from the public record.