Specifiers Are Asking AI First
What an assistant can read on your website now decides what it says when an architect asks.
An architect scoping a hardware set for a project now often opens an AI assistant before opening a manufacturer's website. The assistant answers from what it can fetch and parse as plain text from a public URL, so the parts of a manufacturer's site that work hardest for human visitors are frequently the parts a machine cannot read.
That gap is where product data goes missing. A manufacturer can hold current test reports, a complete guide specification, and ten years of environmental documentation, and still return nothing when an assistant is asked a direct question.
What an assistant can actually read
An AI assistant reads selectable HTML text on a public page. Everything else is invisible to it: text baked into an image, a specification served as a Word file, a CAD detail, a scanned PDF, a configurator that assembles its output in the browser, and anything sitting behind a lead form or a login.
A scanned PDF is, to an assistant, a picture of a specification.
This is a formatting problem before it is a marketing problem. The information is usually already written. It is stored in containers machines cannot open.
Where product information goes missing
Four patterns account for most of what we find in the site audits SPC performs for members. No company is named here, and the patterns are common enough that most manufacturers will recognize at least one.
- Specifications delivered only as downloads. On one member site, every product configuration offered four files: a specification in .docx, a detail PDF, a DWG, and a cut sheet. The cut sheet extracted cleanly as text. The specification, the single document an architect most needs, served as binary our fetch tool could not parse. DWG files are unreadable to crawlers by nature.
- Guide specifications and product guides behind lead forms. A gated asset cannot be cited, quoted, or summarized by an assistant. Whatever it earns in contact records, it contributes nothing to the answer a specifier receives.
- Environmental documentation that exists and was never published. One manufacturer we audited has maintained environmental product declarations for years and renewed them on schedule. Its public site carries no page saying so. An architect who asks an assistant whether that company holds a current EPD receives no answer from the company itself.
- Facts that live only in images or menu labels. Warranty terms, lead times, acoustic ratings, and finish availability are among the most requested and least published items on a manufacturer site. A menu item reading Warranties with no readable page behind it answers nothing.
Three lines in robots.txt worth checking today
Most manufacturer sites that block AI crawlers do so by accident. Three settings account for the majority of it.
Disallow: /*?removes every URL carrying a query string, including filtered product views, on-site search results, and paginated indexes. One line, and an expensive one.- A long
Crawl-delaystarves crawlers working from a limited fetch budget. Several ignore the directive entirely, so the setting costs reach without buying protection. - A missing
Sitemap:line leaves crawlers to find pages by following links. Serve a sitemap that parses as plain XML on a direct fetch. A sitemap that returns unreadable binary is a failed sitemap.
Name AI crawlers explicitly, and verify the current user-agent strings at the moment you write the file, since operators add and rename them.
Two categories are worth separating. Retrieval crawlers fetch a page to answer a live user question and usually cite the source. Training crawlers collect content for model training. Allowing the first and deciding separately on the second is a legitimate position, and robots.txt supports it.
The ten-question test
Write down the ten questions a specifier asks before naming your product in a section, then find the public URL that answers each one in its first two sentences. Every question without a URL is a gap, and the list of gaps is your work order.
A workable starting list for openings, specialties and electronic safety and security products:
- What is the product, and what generic category does it belong to?
- What fire, smoke, acoustic, and security ratings does it hold, with each standard named and dated?
- What are the sizes, materials, finishes, and tolerances?
- What does the full specification section say?
- Is there a current environmental product declaration or health product declaration?
- What are the warranty terms?
- What are current lead times?
- Which sustainability credits can the product contribute toward?
- Where is it manufactured?
- Who sells and services it in a given territory?
Run the test with a plain HTTP client. A browser executes scripts and renders images, which conceals the exact failures you are looking for.
What this has to do with getting specified
A product reaches a specification when a specifier can verify it quickly. Verification now runs partly through assistants, and an assistant that cannot verify a claim answers from whatever it can reach: a trade article from a decade ago, a distributor's summary, a page someone else published better.
SPC's spec writer members develop guide specifications with manufacturers. The same rule governs the finished document. A guide specification that exists only as a Word file behind a form can be quoted only by people already on your mailing list.
The same holds across openings, specialties, and electronic safety and security. All three carry the ratings, dimensions, and compliance claims that assistants are asked to retrieve, and all three commonly store them in downloads.
Questions we get
- Does allowing AI crawlers give away my content?
- Retrieval crawlers fetch a page to answer a live user question and usually cite the source, which sends the reader back to you. Training crawlers collect content for model training. Each is addressed separately in robots.txt, so the two decisions can be made on their own terms.
- Should I remove my PDFs?
- Keep them. Publish an HTML page carrying the same facts and offer the PDF as the printable version. PDFs also need a real text layer, so verify that text extracts from every published file.
- We just rebuilt the site. Does that mean we pass?
- A site that passed at launch drifts as pages get gated, migrated to PDF, or replaced with rendered graphics. Re-run the ten-question test quarterly and after any significant content change.
- Is this the same as SEO?
- It overlaps with SEO and it is a separate test. A page can rank well in a search engine and still be unreadable to an assistant, most often because the facts live in a download.
Most manufacturers we audit already hold the documentation. What they lack is a public page carrying it in readable text. The test costs an afternoon and returns a precise list of which answers a specifier can get today.
Related program: AI Crawler Audits