Published September 17, 2026 · Last updated September 17, 2026 · Bot names change — re-checked quarterly.

Are AI bots accidentally blocked from my contractor website?

Before you spend a dollar or an hour on anything else: rule out the dumbest reason first. A surprising number of contractor websites quietly block the very bots that feed AI answers — and the owner has no idea. This check takes two minutes and it's free.

What AI bots actually are

The AI companies send automated crawlers across the web to read pages, the same way Google's crawler reads pages for search. The difference: these bots feed AI answers and citations, not search rankings. The industry-known names include GPTBot and OAI-SearchBot (OpenAI's crawlers). Other AI companies run their own.

Here's the key point: if these bots can't read your site, the AI has no first-hand material from you to quote. It will quote your competitor's site instead — or quote a directory about you, which you don't control. Your own website is the single biggest source of AI citations — Yext reports 44% of AI citations come from websites — so being unreadable is the most expensive invisible problem you can have.

The three ways contractors block them by accident

1. A robots.txt file that says "keep out"

Every website can carry a small file at yoursite.com/robots.txt that tells crawlers which pages they're allowed to read. The classic accident: a web developer builds your new site on a staging server, blocks all crawlers during the build with a "Disallow: /" rule (totally correct for staging), and then copies everything — including that rule — to the live site. Now every crawler, AI or otherwise, is told to stay out. Some sites also carry rules that specifically disallow AI bot names, added by a developer or a plugin default the owner never saw.

2. Cloudflare or a firewall's bot-fighting settings

Many contractor sites sit behind Cloudflare or a security firewall. Those tools have "bot fight" modes that challenge or block automated visitors. They're tuned to stop malicious bots, but aggressive settings can swat legitimate AI crawlers too. If your site uses Cloudflare's bot-fighting features or a strict firewall, your AI visibility may be collateral damage.

3. WordPress security plugins

WordPress security plugins sometimes ship with crawler-blocking defaults, or an "AI content protection" feature that sounds like it's protecting you and is actually hiding you. If your site runs a security plugin, check whether it blocks AI crawlers — the setting is often one toggle buried three menus deep.

Blocking training is not the same as blocking answers

This is the distinction most advice skips, and it matters. AI companies generally operate (or document) two separate kinds of crawling:

• Training crawlers — collect web content to train future AI models.
• Retrieval crawlers — fetch live pages so the AI can quote and cite them in answers right now.

Blocking one does not require blocking the other. If your concern is "I don't want my content used to train AI," you can block the training crawlers while allowing the retrieval crawlers — and still show up in AI answers. A blanket "block all AI bots" rule throws away your citations to solve a training-data worry. Make the two decisions separately.

The 2-minute check

1. Open yoursite.com/robots.txt in a browser (replace "yoursite.com" with your domain). Read it. If you see Disallow: / applied broadly — or AI bot names next to "Disallow" — you've found the problem.
2. Look for AI bot names specifically. Search the file for names like GPTBot and OAI-SearchBot. Allowed looks like Allow: / under the bot's name (or simply no rule mentioning it). Blocked looks like Disallow: /.
3. Check your firewall. If you use Cloudflare, look at the bot-fighting and WAF settings, or ask whoever manages your site: "are we blocking AI crawlers?"
4. Check your security plugin. In WordPress, open your security plugin's settings and search for "AI," "crawler," or "bot" options.

If anything above is blocking AI crawlers and you want to show up in AI answers, the fix is to allow the retrieval crawlers. That's usually a one-line robots.txt edit or one toggled setting — your web person can do it in minutes. After the fix, give it a few weeks: crawlers need to revisit before the AI's answers change.

Allowing AI bots won't make your site less safe

A worry we hear: "if I let bots in, am I opening the door to hackers?" No. Allowing a crawler to read your public pages is not the same as giving it access to anything private. Your contact forms, your admin login, your customer data — none of that becomes reachable because GPTBot can read your water-damage page. Crawlers only see what a visitor sees. Keep your normal security (strong passwords, updates, backups) and let the legitimate crawlers read the public content.

What to do this week

1. Run the 2-minute check above. Right now — it's free.
2. If you find a block, fix only that block. Allow AI retrieval crawlers; leave everything else alone.
3. Then move on to the content. Being crawlable just gets you in the door — here's the full diagnosis of what the AI needs to find once it's inside.