Ads.txt & Robots.txt Setup Guide

To ensure your ad revenue stays protected, you need a two-pronged approach: the ads.txt file must be formatted correctly, and your robots.txt must explicitly permit crawlers to see it.

1. The Proper robots.txt Configuration

Ad crawlers (SSPs) cannot authorize your site if your robots.txt blocks them. Add these lines to ensure universal access:

User-agent: *
Allow: /ads.txt

Tip: Always place the Allow: /ads.txt rule above any Disallow rules to ensure it takes precedence.

2. The ads.txt Technical Checklist

Once the bots can reach the file, the content inside must be perfect. Use this table to audit your file:

Validation Step	Requirement	Why it matters
Location	yourdomain.com/ads.txt	Must be in the root directory, not a subfolder.
HTTP Status	200 OK	A 404 (Not Found) or 403 (Forbidden) stops revenue.
Syntax	domain, pub-ID, type, [cert-ID]	e.g., google.com, pub-000, DIRECT. No typos allowed.
Format	Plain Text (.txt)	HTML or PDF versions will be ignored by crawlers.
Encoding	UTF-8 (without BOM)	Prevents "ghost characters" from breaking the file.

3. How to Verify Everything is Working

Browser Check

Visit yourdomain.com/ads.txt.
If you see your publisher IDs in plain text, the file is live.

Redirect Check

Ensure your URL doesn't redirect to another page. It should go straight to the text.

• Use a tool like Redirect Checker or your browser's Inspect tool (Network tab).
• You want a 200 OK status.
• If you see a 301 or 302 redirect, make sure it eventually lands on the correct file.
• If it leads to a login page or a 403 Forbidden error, crawlers will ignore it.

Format Check

Open the file with Notepad++ and check:

• Format correctness
• Encoding (UTF-8 without BOM)

Only straight quotes (") are allowed. Curly quotes will break the file.

Search Console

If you use Google Search Console, use the Robots.txt Tester to "Fetch" the /ads.txt path.
It will tell you immediately if a rule is blocking it.

The Wait

Even with a perfect setup, ad networks like AdSense or Rubicon can take 24 to 48 hours to update their dashboards.

Common Red Flags to Watch For

• Curly Quotes:
  If you copied your ID from a Word document, you might have “smart quotes” instead of "straight quotes". This will break the file.

• Subdomains:
  If your ads are on blog.mysite.com, you usually need an ads.txt on the root mysite.com as well, or a specific redirect set up.