Technical GEO: open your site to AI crawlers
I find and fix what keeps AI systems and search engines from reaching your content: blocked crawlers, text that only appears after JavaScript runs, firewall rules that answer bots with a 403, markup that contradicts the page. You get a technical report, a spec for your developer and a re-check log – or I make the fixes myself.
The second gate breaks most often. Usually nobody on the team knows.
When does technical GEO come first?
Usually right after an audit, when it shows that AI systems cannot get to your pages in the first place.
Crawlers are blocked
Cloudflare, a WAF or bot protection answers GPTBot or PerplexityBot with a challenge page or a 403. For those systems, your site does not exist.
Text loads only through JavaScript
The initial HTML is nearly empty, and a script loads the content afterward. Some AI crawlers never run scripts, so they see a blank page.
Markup contradicts the page
The schema.org markup lists prices and services that the visible page does not show. That is worse than having no markup at all.
Sections are not indexed
Some pages are disallowed in robots.txt, return the wrong status code or point their canonical tag at another page.
Duplicates and redirect chains
The same content lives at several URLs or sits behind chains of redirects. Its signals are split between the copies, and a model may quote an outdated one.
No single page with the facts
Your company details are scattered across the site, and there is no one URL where a model can take them all at once.
What goes into
technical GEO?
AI crawler access
- robots.txt rules for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended and Bingbot
- Test requests with crawler user agents: status code, headers and the HTML that comes back
- Cloudflare, WAF and bot protection settings that block AI systems without anyone noticing
- Training and search crawlers handled separately: you can opt out of training and stay in ChatGPT search
- Rate limits and server timeouts
Indexing and status codes
- robots.txt and sitemap.xml checked against the real site structure
- Canonical tags, noindex, hreflang and URL parameters
- Status codes, redirect chains and loops
- Duplicates and orphan pages with no internal links
- Coverage reports in Google Search Console and Bing Webmaster Tools
Rendering
- Initial HTML compared with the fully rendered page
- Key text, prices and contacts present in the server response
- Lazy-loaded blocks, tabs and accordions that hide content
- Server-side rendering advice for priority templates
Structure and semantics
- One H1 per page and a consistent heading hierarchy
- Semantic tags: header, nav, main, article, section, footer
- Extractable blocks: definitions, tables, lists of terms, FAQ
- Internal links as standard anchor links with an href
Structured data (schema.org)
- Organization or Person, WebSite and BreadcrumbList on every template
- Service, Product, Article and FAQPage where the page has matching visible content
- Markup values checked against the page text, mismatches removed
- Validation in Schema Markup Validator and Google Rich Results Test
llms.txt and machine-readable facts
- An llms.txt file: a plain-text map of your site and key facts for language models
- A single facts page: name, legal entity, contacts, services, prices and dates
- The same facts in markup and in human-readable text
- Links to verified external profiles
Speed and server response
- Server response time for regular visitors and for crawler requests
- Core Web Vitals on key templates: LCP, INP, CLS
- Heavy scripts and images that delay the main text
Re-check
- The same checks run again after the fixes
- A before-and-after log for every item
- A list of what remains on your team’s side
How does the project run?
Access
The site, Google Search Console, Bing Webmaster Tools and analytics. 1 day.
Technical audit
Crawl, rendering, crawler access and markup checks. 2–3 days.
Report and spec
Issues ranked by priority and a spec for your developer. 1 day.
Scope
You choose what I fix, what your developer fixes and what can wait.
Implementation
If you chose implementation, I make the agreed fixes and release them in batches, testing each one.
Re-check
I run the same tests again and log the results. 1 day.
What do you get?
The report and the spec are written so any developer can pick them up.
Technical report
Every issue with its URL, impact, effort and fix priority.
Developer spec
What to change and how, with code examples and acceptance criteria for each item.
Schema.org blocks
Ready markup for your templates, matched to the visible text of the pages.
Crawler rules for robots.txt
Rules for each AI crawler, written to match the training and search policy you choose.
llms.txt draft
A ready file with your site map and key facts, to publish at the site root.
Redirect and canonical map
Included when duplicates turn up or URLs need to move.
Re-check log
The status of every item before and after the fixes.
Report walkthrough
A call with you or your developer, 40–60 minutes, recorded.
Your developer can do half the fixes in a day. The spec says which half.
Timeline and format
- Audit and spec – 5 business days from when I get access
- With implementation – up to 10 business days, depending on scope
- Fixes go live in batches, each one tested
- The re-check is included in both formats
What I need from you
- Access to the site, or your developer’s contact
- Google Search Console, Bing Webmaster Tools and analytics
- Access to Cloudflare or your CDN and bot protection panel, if you use one
- A staging site, if changes cannot go straight to production
Not included
- Redesign, site migration or a move to another platform
- Complex server work beyond the agreed estimate – your developer does it from my spec
- Rewriting page content – that is page optimization for AI search
How is the result measured?
Technical changes can be checked directly: before and after show up in server responses and in Search Console reports.
Crawler access
Which AI crawlers get a 200 response and the full HTML, before and after.
Content availability
Share of key pages that return their main text without running scripts.
Markup accuracy
Share of templates with valid markup that matches the visible text.
A Telegram store builder named first of 8 in ChatGPT
The site had no llms.txt, and its pricing facts existed only in the visual layout. After the crawler access fixes, extractable facts and a new llms.txt file, ChatGPT now describes the service in the same words as the file. Russian-speaking market.
What usually comes next?
AI visibility audit
If only the technical side has been checked so far, measure visibility itself.
$490 →Page optimization for AI search
Once crawlers can read the pages, the pages need answers worth quoting.
from $90 per page →AI visibility monitoring
So fixes do not roll back unnoticed with the next site release.
from $290/mo →Related reading
AI crawlers on your site: how to check GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot
A "GPTBot" line in the log proves little until the IP matches the official range and you know what the server returned. Here is the four-step check I use, and what Cloudflare's September 2026 change means if your site sits behind it.
Read →What is llms.txt, and how do you create, add and check it?
llms.txt is a proposed Markdown file that gives AI agents a short, curated map of your site. I explain the v2 format, how to build and check the file, and what it can and cannot do for SEO and GEO.
Read →Questions about technical GEO
Do you edit the site yourself or hand a spec to a developer?
Do you need server access?
My site runs on Wix, Squarespace or Shopify. What can you change there?
Is this the same as a technical SEO audit?
How do you check that AI crawlers get the content?
Should I block AI crawlers from training on my content?
Does llms.txt make a difference?
Discuss technical GEO
Send your website address. I will check what the server returns to AI crawlers and tell you within one business day whether a fix list is enough or you need implementation.
I reply within one business day. For a larger project, fill in the project brief.