# OAI-AdsBot and OAI-SearchBot: allowing OpenAI's crawlers for ChatGPT ads

Source: https://gptads.ae/guides/openai-crawlers/
Author: Ahmad Alshareefi, Big AL Consulting, Dubai
Updated: 2026-11-15. Checked against OpenAI documentation: 2026-11-15

> Which OpenAI crawlers ChatGPT ads needs, the robots.txt lines to use, and how to fix firewall, CDN and bot protection blocks step by step.

## The short version

ChatGPT ads landing pages have to be reachable by OpenAI's own crawlers before an ad can pass review. You must allow OAI-AdsBot, and OpenAI recommends allowing OAI-SearchBot as well. Most blocks come from one of three layers, robots.txt, firewall or bot mitigation, and human verification checks, and the fix is almost always to allowlist the crawler at that layer rather than ask support for an exception.

## Why OpenAI crawls your landing page at all

OpenAI uses crawlers to validate the safety of web pages submitted as ads on ChatGPT. When you submit an ad, OpenAI may visit the landing page to confirm it complies with its policies. It may also use content from the landing page to judge when the ad is relevant to show. A landing page the crawler cannot reach is a landing page OpenAI cannot validate, regardless of how the page looks to a real visitor.

This makes crawler access a launch dependency, not an optional technical detail. A perfectly written ad, with accurate copy and a compliant offer, will still fail review if the landing page it points to returns an error, a challenge screen or a blocked response to OAI-AdsBot. Because the check happens automatically during ad review, most advertisers only discover a block exists after a rejection, which is why it is worth confirming crawler access before the first submission rather than after.

## Which crawler you actually need

| Crawler | Requirement | What it is for |
| --- | --- | --- |
| OAI-AdsBot | Required | Landing page validation and ad review. This is the one to prioritise |
| OAI-SearchBot | Recommended | Helps OpenAI understand public web content, and is also used to crawl product feed image URLs |

For product feed campaigns, also allow OAI-SearchBot to crawl the product image URLs specifically. OpenAI notes that a 403 response from the image host's firewall, CDN or bot protection can block feed processing even when the main site is fully accessible.

OpenAI's ad creation guide is stricter on landing pages than the crawler article: it says all landing pages "must not block OpenAI user agents (OAI-AdsBot and OAI-SearchBot)". In practice, allowing both is the safe setting.

## OpenAI's five step check

By our 15 November 2026 check, OpenAI had rewritten this part of its crawler article as a checklist to run before submitting ads, or after changing your website, and to share with your web developer, host or security team.

Check the exact link used in your adOpen the complete ad URL, tracking parameters included, follow the redirects and confirm the final page shows what you advertise. If you use a tracking link, check where it sends people on different devices and in different locations. OpenAI's point: "A working homepage does not confirm that the link in your ad works."
Allow the relevant OpenAI crawlersAllow OAI-AdsBot, which is required, and OAI-SearchBot, which is recommended. Check robots.txt for the landing page and any required assets, on each host involved in a redirect, including subdomains and third party destinations.
Check your security settings"A robots.txt rule alone may not be enough." Confirm verified OpenAI crawler requests reach the page and its content without signing in, a CAPTCHA or a browser challenge, on the destination host as well as the starting domain. If access is denied, use the failed request's time and request ID to find it in your logs.
Check the page the crawler receivesA page that opens in your browser may not be what the crawler receives. See the table below.
Check again after making changesRecord what changed and when, retest the exact ad URL and its redirects, and for a large group of ads start with a few representative URLs. Then check the ads in Ads Manager; fixing crawler access does not guarantee approval.

| What the crawler gets | What OpenAI says to check |
| --- | --- |
| 403 / Access denied | Security rules on the host that denied the request, which may be a redirected subdomain rather than your main site |
| 404 / Page not found | The missing page or the redirect destination, including what a tracking link sends to on each device or location |
| 429 / Too many requests | Rate limits and bot protection events; spread large uploads over smaller batches |
| A blank, incomplete, login or challenge page | That the real content and required assets are reachable; a successful HTTP response alone does not prove the content loaded |

If you still need support, OpenAI asks for the ad account and ad IDs, the exact ad URL and its final destination, the error or a screenshot, when it failed with the time zone, and what your website team checked or changed.

## The robots.txt lines

Review your robots.txt file and confirm both crawlers are explicitly allowed on the relevant pages and paths. The earlier version of OpenAI's article printed this example, and it remains the plain way to do it:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: OAI-AdsBot
Allow: /
```

OpenAI's earlier version noted that its crawlers respect robots.txt and that crawling stops if access is disallowed there. It is the first and simplest layer to check, and it is worth confirming directly rather than assuming an existing robots.txt already covers it, particularly on a site that blocks crawlers broadly by default and allowlists specific ones.

## The three layers that block a crawler

Most websites have more than one layer of protection between the public internet and the page itself. Work through each one in order.

1. **robots.txt.** Confirm OpenAI's crawlers are explicitly allowed on the relevant paths, using the lines above as a template.
2. **Web protection and bot mitigation.** Services such as Cloudflare, Akamai and similar providers defend against scraping and unauthorised traffic, and can mistake a legitimate crawler for automated abuse, often returning a 403 Forbidden response. Review your firewall configuration and allowlist OpenAI's crawler traffic, ideally based on the published user agent names, and have your infrastructure team inspect bot mitigation rules that could be producing false positives.
3. **Human verification and anti bot logic.** CAPTCHAs, JavaScript challenges, behavioural analysis and session validation can all block an automated crawler even after it clears the first two layers, since OpenAI's crawlers are themselves automated systems. Review any such logic on the landing page and exempt OpenAI's crawlers where appropriate, again ideally by user agent.

## A note on stable IP ranges

Some security systems want crawler traffic to originate from stable, published IP ranges before they will reliably allowlist it. Because crawler infrastructure can change over time, OpenAI advises against relying solely on short term IP observations pulled from logs. Instead, validate traffic using a combination of user agent identification, verified bot programs where your provider supports them, firewall allowlists, robots.txt behaviour and provider level bot verification systems. If a stable IP list is genuinely required, OpenAI points to openai.com/searchbot.json and openai.com/adsbot.json as the reference.

## A note on rate limiting

Large batch uploads or a sudden spike in crawler traffic can trigger automated rate limiting or bot protection on its own, separate from any deliberate block. If you suspect this is happening, have your engineering team review HTTP response codes, especially 429 Too Many Requests, alongside firewall or CDN logs, bot mitigation events, request throttling rules and traffic analytics around the time of the crawl attempt. Uploading ads over a longer period in smaller batches can also reduce the chance of triggering it in the first place.

## A note on Cloudflare specifically

OAI-AdsBot is officially verified and allowlisted by Cloudflare. If your site sits behind Cloudflare and a crawler block still occurs, the cause is more likely a custom firewall rule, a page rule, or a bot management setting layered on top of Cloudflare's own allowlist, rather than Cloudflare blocking the crawler by default.

### Reading the layers as a single checklist

Because the three layers stack on top of each other, a page can pass one check and still fail the next. Working through them as a single checklist, rather than fixing the first thing you find and assuming the rest is fine, saves a second rejection cycle.

| Layer | Common symptom if blocked | Who usually owns the fix |
| --- | --- | --- |
| robots.txt | Crawl stops immediately, before any HTTP request completes | Whoever manages the site's root files or CMS settings |
| Firewall, CDN or bot mitigation | A 403 Forbidden response, often intermittent rather than constant | Infrastructure, hosting provider or CDN administrator |
| Human verification, CAPTCHA, JavaScript challenge | The page loads for the crawler but content cannot be read, or the request is challenged | Whoever implemented the anti bot or session logic on the page itself |

A page that fails at robots.txt never reaches the second or third layer, so fixing robots.txt first and re-testing before touching the firewall is the more efficient order, even though all three ultimately need to be correct.

## What your engineering team should check first

OpenAI's own troubleshooting order is short: confirm the landing page returns a successful HTTP response to OAI-AdsBot, confirm robots.txt allows the relevant path, and confirm that no WAF, CDN, bot mitigation, JavaScript challenge, CAPTCHA, authentication requirement or geo rule is blocking automated access. Working through these in this order avoids chasing a firewall issue when the actual block is a single disallow line in robots.txt, or the reverse.

## What is not a supported landing page

Use a directly reachable web landing page wherever possible. OpenAI flags app store links, deep links, documents, and destinations that require an app, a login, region specific access, or an unsupported redirect as potentially not providing enough crawlable content for validation or review. If your funnel currently sends ChatGPT ads traffic to one of these, plan for a standard web landing page as the ad destination instead, even if the eventual conversion still happens inside an app.

## What to do after you fix the block

Do not expect a manual bypass from support. OpenAI states this directly: make the landing page actually crawlable by fixing robots.txt, WAF, CDN, bot mitigation, authentication or rate limiting, since there is no workaround that skips crawler validation. Once the fix is in place, re-upload or resubmit affected ads if their status does not update on its own. For a bulk upload, submitting smaller batches can reduce the chance of hitting a rate limit or bot protection trigger again while your team confirms the fix holds.

## Common mistakes and what goes wrong

- **Testing only in a browser.** A page that loads perfectly for a human tester can still block OAI-AdsBot if a CAPTCHA, JavaScript challenge or session check sits in front of it. Confirm crawler access specifically, not just human access.
- **Fixing robots.txt but leaving the firewall untouched.** All three layers, robots.txt, bot mitigation and human verification, have to be checked. Clearing one does not guarantee the others are not still blocking the crawler.
- **Forgetting the product feed image host.** The main site can be fully open while the image CDN, often a different domain or subdomain, still returns a 403 to OAI-SearchBot and silently blocks feed processing.
- **Relying on an old IP allowlist.** Crawler infrastructure can change, and OpenAI is explicit that short term log based IP observations are not a reliable long term allowlist strategy.
- **Assuming support can override a block.** There is no manual bypass. The page has to genuinely become crawlable, and ads may need to be resubmitted afterwards even once it is.
- **Uploading a large batch immediately after fixing a block.** A sudden spike right after a fix can itself trigger rate limiting, which looks identical to the original problem. Consider smaller batches while confirming the fix is stable.

## Why this shows up as an ad rejection

In the Ads tab, an ad blocked by a crawler issue typically shows a status of Not serving, with the reason visible on hover, and common causes listed by OpenAI include ad copy or landing pages that do not align with its ad policies, or landing pages that block OpenAI's user agents or IP addresses. If you see a rejection with no obvious policy problem in the copy itself, a crawler block on the landing page is one of the first things worth ruling out, alongside the account and billing checks covered in the [ChatGPT ads not showing guide](/guides/chatgpt-ads-not-showing/). Getting this right the first time also matters at account setup, since a rejected first ad can delay the review timeline covered in the [account setup guide](/guides/ads-manager-account-setup/).

## A UAE hosting note

Many UAE businesses run their site behind a regional CDN or a hosting provider's own bundled firewall, sometimes on top of Cloudflare, sometimes instead of it. Do not assume a UAE specific configuration is the cause of a block, since OpenAI does not publish any region specific crawler rules. Treat it as the same three layer check as anywhere else, starting with robots.txt, then the firewall or CDN, then any human verification logic, and confirm with your hosting provider directly whether OpenAI's crawler user agents are allowlisted by default or need to be added.

If your landing page sits on a shared hosting plan rather than a dedicated server, the bundled security layer is often managed by the hosting provider's support team rather than being configurable from your own dashboard. In that case, the practical path is a support ticket to the host asking them to allowlist the OAI-AdsBot and OAI-SearchBot user agents specifically, referencing OpenAI's published crawler guidance, rather than trying to locate a firewall setting that may not be exposed to you at all.

## Quick answers

**Which OpenAI crawler is required for ChatGPT ads?**
OAI-AdsBot. OpenAI states it is required for ChatGPT ads landing page validation and review, and advertisers should prioritise it. OAI-SearchBot is recommended in addition, because it may help OpenAI understand public web content.

**What does my robots.txt need to say?**
It must allow OAI-AdsBot on the landing page and any assets it needs, and OpenAI recommends allowing OAI-SearchBot too, on every host in the redirect chain. An earlier version of OpenAI's article printed the example User-agent: OAI-SearchBot, Allow: / and User-agent: OAI-AdsBot, Allow: /. The current version adds that a robots.txt rule alone may not be enough, since a firewall or CDN can still block the crawler.

**What is OpenAI's checklist for crawler access?**
Five steps: check the exact ad link with its tracking parameters and redirects, allow the relevant crawlers on every host, check your security settings, check what the crawler actually receives, such as a 403, 404, 429 or a blank or challenge page, and check again after any change.

**Why was my ad rejected even though my landing page works fine in a browser?**
A landing page that blocks OpenAI's crawlers, through robots.txt, a firewall, bot mitigation or a CAPTCHA, can fail ad review even though a human visitor sees the page normally, because the crawler is what validates the page, not a human tester.

**Can OpenAI support manually bypass a crawler block?**
No. OpenAI states not to rely on a manual bypass. The landing page has to actually be made crawlable for OAI-AdsBot, by fixing robots.txt, WAF, CDN, bot mitigation, authentication or rate limiting, and ads may need to be re-uploaded after the fix.

**Does Cloudflare block OpenAI's ad crawler by default?**
OpenAI states that OAI-AdsBot is officially verified and allowlisted by Cloudflare, though other bot protection layers on top of Cloudflare, such as custom firewall rules, can still block it.

**Should I allowlist OpenAI's crawlers by IP address?**
OpenAI advises against relying solely on short term IP observations from logs, since crawler infrastructure can change. If a stable IP range is required, it points to openai.com/searchbot.json and openai.com/adsbot.json, combined with user agent identification and provider level bot verification.

**Are app store links or login protected pages supported as ChatGPT ads landing pages?**
OpenAI recommends a directly reachable web landing page whenever possible. App store links, deep links, documents, or destinations needing an app, login, region restricted access or unsupported redirects may not provide enough crawlable content for validation.

## Sources

- OpenAI Help Center, Advertiser Guidance for Allowing OpenAI Web Crawlers: https://help.openai.com/en/articles/20001243-advertiser-guidance-for-allowing-openai-web-crawlers
- OpenAI Help Center, Create Ads for ChatGPT Ads: https://help.openai.com/en/articles/20001212-create-ads-for-chatgpt-ads
- OpenAI Help Center, Troubleshooting Common Issues: https://help.openai.com/en/articles/20001217-troubleshooting-common-issues
