Best seo robots txt for google search
The "robots.txt" file is an essential tool in search engine optimization (SEO) that helps Google and other search engines crawl your website more efficiently. Here is an example of a basic robots.txt file that you can use for Google search:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Disallow: /author/
Disallow: /tag/
Disallow: /archives/
Allow: /wp-content/uploads/
Allow: /sitemap.xml
The above robots.txt file:
- Allows all user agents to crawl the website (User-agent: *)
- Disallows indexing of WordPress admin pages (Disallow: /wp-admin/)
- Disallows indexing of WordPress core files (Disallow: /wp-includes/)
- Disallows access to XML-RPC (Disallow: /xmlrpc.php)
- Disallows indexing of WordPress plugins (Disallow: /wp-content/plugins/)
- Disallows indexing of WordPress cache files (Disallow: /wp-content/cache/)
- Disallows indexing of WordPress theme files (Disallow: /wp-content/themes/)
- Disallows indexing of trackback pages (Disallow: /trackback/)
- Disallows indexing of RSS feed pages (Disallow: /feed/)
- Disallows indexing of comment pages (Disallow: /comments/)
- Disallows indexing of category pages (Disallow: /category/)
- Disallows indexing of pages with trackbacks (Disallow: */trackback/)
- Disallows indexing of pages with RSS feeds (Disallow: */feed/)
- Disallows indexing of pages with comments (Disallow: */comments