What is robots.txt?
robots.txt is a plain text file that tells search engine bots which pages or files they can or cannot request from your site. Moreover, it helps control crawl budget and prevents indexing of duplicate or sensitive content.
Sample robots.txt Template
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap.xml
Explanation:
- First, User-agent: * — Applies to all bots.
- Next, Disallow: — Blocks access to the wp-admin folder.
- Then, Allow: — Permits AJAX calls from that folder.
- Finally, Sitemap: — Declares your XML sitemap URL.
What is a Sitemap?
A sitemap is an XML file that lists all the important URLs on your website. In addition, it helps search engines discover and crawl your content more efficiently.
Sample Sitemap.xml Structure
https://example.com/
2025-07-02
1.00
https://example.com/blog/
2025-06-25
0.80
✅ Best Practice: Always keep your robots.txt file in your root directory, and furthermore, link to your sitemap to guide search engines effectively.
Where to Submit Sitemap
- First, Google Search Console: Find it under the “Sitemaps” tab.
- Next, Bing Webmaster Tools: Submit your sitemap to improve crawling on Bing.
- Finally, use plugins like Yoast SEO or Rank Math: These auto-ping your sitemap whenever updates are made.
📚 Helpful Articles
