Major Labs · research practice
Archive 001 · The Front Door
Major Labs · research practice
001Archive 001 · Agentic commerce · The front door

I knocked on 92 storefronts. 28 would not show me the rules.

Checked. On September 26, 2026, 28 of 92 large US retail storefronts withheld robots.txt from an agent that said who it was. On September 20 it was 29. Of the 64 that served the file, 16 name any AI crawler or agent. A readable homepage came back from 49. Sixteen publish an llms.txt, three publish a UCP profile, and none has an A2A agent card.

Sample: the NRF Top 100 Retailers 2026, minus the eight with no single storefront. One US cloud address, every Saturday. A refusal cannot say why. A timeout counts as no answer. File counts are undercounts. No retailer is named from scan data.

Scroll to walk the street
The Front Door · BookletCharlie Major · Runs of 2026-09-20 and 2026-09-26
Provocation · What the door does when an agent knocks

When Amazon cut off Meta's Muse agent, it gave a reason that sounded like a rule: agents that buy on a customer's behalf "should operate openly and respect service provider decisions about whether or not to participate."

I read that and wanted to know what the rule was at the other large retailers in America. Not what they said about AI in an interview. What their sites do when an agent knocks.

There was nowhere to look it up. No survey asked the sites. The robots.txt trackers count publishers, not stores. The standards bodies writing agent-identity specs had no count of how many merchants would use one. A decision that affects every shopper with an assistant was being made, one site at a time, by a default nobody had read.

So I asked the sites directly. Ninety-two of them, six polite questions each, the same way every week. Can an agent read the rules? Does the agent get a page? Is anything published for the agent to find? Rules, enforcement, intent. Those three are the whole door.

A third will not show a self-identified agent their robots.txt. Roughly half serve a readable homepage to a client that is not a browser. Three publish a profile that tells an agent what it may do, down to checkout.

Between the two runs, 89 of 92 sites gave the same answer on the rules question and 89 of 92 on the homepage. The three that flipped are the noise floor. The aggregate is stable, and a stable measure is the only kind in which a change means something.

The frame

The merchant does not know what its bot manager is refusing. The agent does not know the rules. The standards body does not know the demand. Somebody should keep the number.

Open the live index
The one file the web sets aside to tell a visitor the rules, and the commonest answer among the storefronts that withheld it. I drew this exchange; it is not copied from any one site.Drawing · Major Labs · request as the scanner sends it
Contents · Eight chapters · Seven data plates · Three documents
    The board

    Evidence, plate by plate

    Every figure here comes from the scan and carries its run date. Counts only. The street behind this page is drawn from the same numbers.

    Method

    Deliberately dull

    The scanner behaves the way a well-mannered agent would. Every step is written down, so anyone with a laptop can reproduce every figure.

    The six requests

    1. /robots.txt, the rules
    2. /llms.txt, a guide written for AI systems
    3. /.well-known/ucp, the commerce profile
    4. A2A agent card, first path
    5. A2A agent card, second path
    6. The homepage, asked for without a browser

    What I never do

    • No login, no form, no cart.
    • No pretending to be a browser or another company's bot.
    • No retry after a refusal.
    • No ranking, and no naming of retailers from scan data.
    • Requests go 1.5 seconds apart, under a user agent that names the project and links to it.

    The limits

    • A refusal cannot tell you why.
    • A timeout is reported as no answer, because a slow site and a deliberate stall look identical from outside.
    • A site that refuses the scanner cannot show it an llms.txt, so file counts are undercounts.
    • A UK home connection sees more refusals than a US data center. That is geography, not agents.
    • One week is worth nothing. The trend is the product.
    Sources

    Where to check me

    1. Major MattersThe Merchant Agent-Readiness Index
    2. Payments DiveAmazon blocks Meta's AI shopper
    3. CloudflareYour site, your rules, new AI traffic options for all customers
    4. CloudflareWeb Bot Auth
    5. GoogleUCP profile guide
    6. Fintech BrainfoodAI Checkout
    7. Major LabsMCP security scoreboard
    8. Major LabsAgent Identity Tracker
    9. NRFTop 100 Retailers 2026
    The booklet · PDF For subscribers · PDF

    Take the whole archive with you.

    The booklet goes to subscribers.

    The booklet is this page set for print: every plate, the method in full, and the run-by-run table. Subscribe to Major Labs Weekly and the PDF is yours to download.

    I write one of these for each study. Subscribers get the PDF, with every plate, the method in full, and the run-by-run table. Add your address and this one is yours.

    Archive 001 · where it lives

    The street is walked again every Saturday.

    The figures in this booklet are two readings of a weekly series. The live index, the open feed and the full method are at majormatters.co/trackers/merchant-readiness. The street itself, which moves as you scroll, is at majorlabs.co.

    Use it. Cite it. Argue with the method; every choice is written down and I will change the ones that are wrong.

    Built by Charlie Major. Major Labs is independent of Mastercard and operates separately from Major Matters. Any opinions are Charlie's own. Format inspired by the research archive pages of Dami Lee and Nollimedia.

    • PlatesSeven, with run dates
    • TablesTwo runs, three vantage points
    • MethodEvery request, every limit

    Major Labs Weekly is one email a week. Unsubscribe in one click. Privacy.