Setting Up robots.txt and sitemap.xml on Your Hosting

Why robots.txt and sitemap.xml Matter for Your Site

Two small files help guide how search engines crawl and discover your website: robots.txt and sitemap.xml. They are small, but they sit right at the intersection of SEO, hosting configuration and long‑term maintainability. A clean robots.txt can reduce compliant crawlers’ requests to low-value URLs, but it does not protect private areas. A well‑structured sitemap.xml can help search engines discover new and updated pages. Misconfigure them, and you can accidentally block crawling of your whole site, delay discovery of new pages, or complicate URL discovery across domains and subdomains.

In this guide, we will walk through practical examples for setting up robots.txt and sitemap.xml on shared hosting, cPanel/DirectAdmin, and VPS servers with Apache or Nginx. We will cover what to allow, what to block, where to serve each file, how to test, and how to adapt your setup for multilingual sites, separate blogs/stores or staging environments. The server snippets illustrate routing choices, not complete configurations for every application. If you are planning a new launch, combine this guide with our new website launch checklist for hosting‑side SEO and performance to plan the broader technical checks.

robots.txt and sitemap.xml in Plain Language

What is robots.txt?

robots.txt is a simple text file placed at the root of your domain, for example:

https://example.com/robots.txt

Search engine crawlers (“bots”) request this file before they start exploring your pages. Inside robots.txt you give rules like:

• Which folders or URL patterns should not be crawled
• Optional crawl‑delay for some bots
• Where your sitemap.xml file lives

It is important to understand that robots.txt is not a security feature. It only tells well‑behaved crawlers what you prefer. Never rely on robots.txt to hide sensitive data; protect those with authentication and proper permissions on your hosting or VPS. If you want a refresher on hosting fundamentals, our article what is web hosting and how domain, DNS, server and SSL work together is a good background read.

What is sitemap.xml?

sitemap.xml is an XML file (or a set of files) that lists important URLs for your site, typically including:

• Each page or post URL
• When it was last modified
• Optional <priority> and <changefreq> hints

Typical location:

https://example.com/sitemap.xml

Search engines use sitemaps to:

• Discover new content
• Find pages not easily reachable via menus or internal links
• Discover URLs for different sections or language versions of your site

Sitemaps do not guarantee indexing, but they can improve discoverability and crawling efficiency, particularly for large sites or pages with few internal links.

Step 1 – Decide Your Crawl Strategy Before Writing a Single Rule

Before opening a text editor, decide what you want bots to see versus what they should ignore. This is where SEO, information architecture and hosting structure meet.

Pages and sections you usually want crawled

• Core public pages: home, category pages, product pages, blog posts, landing pages
• Pagination that adds value (e.g. /blog/page/2/) if your SEO strategy relies on it
• Language or regional versions, depending on your subdomain vs subdirectory choice for SEO and hosting

URLs to consider restricting from crawling

• Admin panels: /wp-admin/, /administrator/, /cp/, custom admin paths
• Internal search or filter URLs with many parameters: ?sort=, ?filter=, ?session=, etc.
• Cart and checkout steps (SEO usually focuses on product/category pages instead)
• Tracking or A/B testing URLs like ?utm_source= or ?variant= (often handled via canonical tags; blocking crawling prevents crawlers from reading those tags)
• Staging or test subdirectories like /staging/, /beta/, /old-site/

These are crawl-control candidates, not an indexing-removal checklist. For public pages that should stay out of search, use a supported noindex meta tag or HTTP response header and allow crawling so the directive can be read. Protect private staging environments with access controls as described below.

Critical warning: robots.txt vs HTTP authentication

If you run a staging site on the same hosting account, never rely only on robots.txt to keep it out of search. Use:

• HTTP Basic authentication over HTTPS (.htpasswd on Apache, basic auth on Nginx), and/or
• IP whitelisting or VPN access

Robots.txt prevents polite bots from crawling; it does not prevent visitors or leaky links from exposing your test environment.

Step 2 – Building a Clean robots.txt (With Real Examples)

Basic structure

The syntax is simple but very strict about spelling and placement. A minimal robots.txt that allows everything and references a sitemap looks like this:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Key points:

• User-agent defines which crawler the rules apply to (e.g. Googlebot, Bingbot). * defines the fallback group for crawlers without a more specific matching group. Google and Bing do not simply add this fallback group to their specific group; keep intended restrictions in each applicable group.
• Disallow followed by nothing means “nothing is disallowed” → full access.
• Sitemap can appear anywhere in the file and can be listed multiple times (for multiple sitemaps).

Typical robots.txt for a CMS site (e.g. WordPress)

For a WordPress site installed at the domain root whose SEO plugin generates /sitemap_index.xml, a starting example is shown below. Adapt the admin and search paths for subdirectory installations, and use your generator’s actual sitemap URL:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap_index.xml

What this does:

• Blocks most of the admin area from crawlers
• Allows admin-ajax.php so some themes/plugins can load content correctly
• Discourages crawling internal search result pages
• Points bots to the main sitemap index generated by an SEO plugin

Blocking parameters or specific bots

If you see certain bots hammering your server or wasting crawl budget on low‑value URLs, you can add more targeted rules.

Request that compliant crawlers avoid these folders, but only if they contain no CSS, JavaScript or other resources needed to render public pages:

User-agent: *
Disallow: /tmp/
Disallow: /cache/

Apply rules only to a specific bot:

User-agent: BadBot
Disallow: /

The example below restricts the lowercase session and sort parameters when they occur first (after ?) or later (after &) in an ordinary query string. Use it only for URLs you intend to exclude from crawling; blocked pages cannot supply their own canonical or noindex tags to the crawler. Google ignores Crawl-delay; Bing supports it. Because the specific Bingbot group does not inherit the general restrictions, they are repeated there. The value 5 is an example, not a recommendation for every site. When combining these snippets into one file, preserve the other restrictions you want in each applicable group.

User-agent: *
Disallow: /*?session=
Disallow: /*&session=
Disallow: /*?sort=
Disallow: /*&sort=

User-agent: Bingbot
Disallow: /*?session=
Disallow: /*&session=
Disallow: /*?sort=
Disallow: /*&sort=
Crawl-delay: 5

What you should almost never do

• Never write Disallow: / for all user‑agents on a live site unless you intentionally want zero crawling.
• Do not block CSS/JS that are needed for rendering; search crawlers need those assets to render pages correctly. Treat this crawl-access check separately from testing your website speed and Core Web Vitals correctly.
• Do not use robots.txt to “hide” passwords, database dumps, backups or logs; those files should not be web‑accessible at all.

Step 3 – Creating sitemap.xml the Right Way

Basic XML structure

A simple sitemap with two URLs looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2025-01-01</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/about/</loc>
    <lastmod>2025-01-05</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.5</priority>
  </url>
</urlset>

Mandatory tags:

• <urlset> root element with proper namespace
• <url> container for each URL
• <loc> canonical URL

<lastmod>, <changefreq> and <priority> are optional hints. Use an accurate last-modified date for each page, or omit <lastmod> if it is unknown. Keep any frequency and priority hints realistic rather than marking everything daily/1.0.

Sitemap indexes for large or complex sites

If your site has more than 50,000 URLs or you want to split content by type (posts, products, categories, languages), you can use a sitemap index file that points to multiple sitemaps:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemaps/sitemap-pages.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemaps/sitemap-posts.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemaps/sitemap-products.xml</loc>
  </sitemap>
</sitemapindex>

In robots.txt you then reference only the index:

Sitemap: https://example.com/sitemap_index.xml

Generating sitemaps on different platforms

WordPress

WordPress supports automatically generated sitemaps:

• Core WordPress sitemap (since 5.5): usually at /wp-sitemap.xml
• SEO plugins (Yoast, Rank Math, etc.) often use /sitemap_index.xml and provide fine‑grained control

On shared hosting or VPS with WordPress, we recommend:

• Use one reliable sitemap source (core or plugin); avoid multiple competing sitemaps.
• Exclude low‑value taxonomies (e.g. tags with little content) from the sitemap via plugin settings.
• Treat database maintenance separately from sitemap setup. Before cleaning wp_options, back up the database and remove only entries whose purpose and disuse you have verified. Our guide on WordPress database optimization and cleaning wp_options/autoload bloat covers this separate maintenance topic.

Custom PHP, Laravel, Symfony or static sites

For custom apps hosted on VPS or shared hosting, you have two main options:

1. Static sitemap.xml generated periodically by a script or build pipeline.
2. Dynamic sitemap endpoint that queries your database and outputs XML on the fly.

Static sitemaps are simpler and lighter on resources; you can regenerate them via a cron job when content changes. Dynamic sitemaps can reflect current database content, but freshness also depends on cache updates. They require careful caching and database indexing (especially on large catalogs or marketplaces; see our article on search infrastructure and hosting choices for large catalog sites for broader architecture tips).

Common sitemap mistakes

• Including URLs that return 404, 410 or redirect (3xx). Every entry should ideally be a 200 OK canonical URL.
• Listing both HTTP and HTTPS versions after a full HTTPS migration; use only one canonical scheme. If you are planning such a migration, read our full HTTP→HTTPS migration guide with 301 redirects and HSTS.
• Exposing staging or private sections through the sitemap even though they are blocked in robots.txt.
• Letting a sitemap exceed either 50,000 URLs or 50MB (52,428,800 bytes) of uncompressed content. Split it before exceeding either limit; gzip does not raise the limit.

Step 4 – Where and How to Place robots.txt and sitemap.xml on Your Server

On shared hosting with cPanel

On most cPanel servers, the public root of your main domain is public_html/. For addon domains or subdomains, they usually have their own document root folders.

1. Before changing files, request the domain’s public /robots.txt URL and its selected sitemap URL. Save the current output and identify whether a CMS, plugin or physical file manages each endpoint.
2. Log in to cPanel.
3. Open “File Manager”.
4. Navigate to the document root of your domain (e.g. /home/username/public_html/).
5. Check for an existing robots.txt and compare it with the public output. If the CMS generates the response, make changes through its supported settings or customization mechanism. An absent physical file does not mean the endpoint is missing.
6. Create a static file at the document root (same folder as index.php or index.html) only for an intentionally managed static setup or replacement. Preserve all required rules and sitemap references from the existing output, supply complete content, and retain appropriate file permissions. A populated file can shadow dynamic output just as an empty one can.
7. Save changes through the chosen source and compare the live response with the intended rules.
8. Ensure your sitemap (static file or CMS‑generated) is accessible, e.g. /sitemap.xml or /sitemap_index.xml. Use the generator’s actual endpoint and do not upload a placeholder over a dynamic route.

Then check in your browser:

• https://yourdomain.com/robots.txt
• https://yourdomain.com/sitemap.xml (or your actual sitemap index URL)

On DirectAdmin or Plesk

The logic is similar: first inspect the public /robots.txt response and the actual sitemap endpoint, then identify whether each is generated by the CMS or served from a physical file. Find the document root (for example domains/yourdomain.com/public_html/ on DirectAdmin) if static files need editing. Manage CMS-generated output through its supported settings or customization mechanism; create or replace robots.txt only as an intentional static deployment that preserves the required rules and sitemap references. Do not infer that the endpoint is missing because no physical file exists. Whether static or generated, robots.txt must be served at the top‑level path for each hostname you want to control. Recheck both public endpoints after saving.

On a VPS or dedicated server with Apache

If you host your site on a VPS or dedicated server from dchost.com using Apache, a static robots.txt can be served from the site’s DocumentRoot. A CMS-generated robots.txt may instead use an application route without a physical file. For a typical VirtualHost:

<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /var/www/example.com/public
    # Other VirtualHost directives omitted
</VirtualHost>

Place static files in /var/www/example.com/public/. The following example belongs in that document root’s .htaccess, not directly inside the VirtualHost above: its patterns have no leading slash. It requires mod_rewrite, permission to use rewrite directives (such as AllowOverride FileInfo), and an allowed FollowSymLinks or SymLinksIfOwnerMatch option. The exceptions apply only when the requested file exists. A dynamically generated robots.txt or sitemap must reach the application’s existing route, so do not create an empty physical file that would shadow it. This front-controller example assumes an application that routes through index.php; keep your framework’s actual rules and test both endpoints:

RewriteEngine On

RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^robots\.txt$ - [L]
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^sitemap(_index)?\.xml$ - [L]

# Illustrative document-root front controller; adapt to your application.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^ index.php [L]

On a VPS or dedicated server with Nginx

The following is a fragment illustrating static-file locations within an existing Nginx server block. It is not a complete PHP site configuration: PHP‑FPM handling and any TLS configuration are omitted and must already be configured before using the PHP fallback shown here.

server {
    server_name example.com;
    root /var/www/example.com/public;

    location = /robots.txt { }
    location = /sitemap.xml { }

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    # PHP-FPM, SSL, etc.
}

The location = /robots.txt { } line selects a static-file location using the configured root; it does not fall through to the application route in the example’s general location. The same applies to the exact sitemap location. If you have a dynamic sitemap endpoint (for example /sitemap.xml served by PHP), preserve the application’s routing to its PHP‑FPM handler rather than using the static-only location shown. Apply the same safeguard to a dynamically generated robots.txt. Adapt the configuration to the application and test both public endpoints.

Multiple domains and addon domains on the same hosting account

Each hostname is treated separately by crawlers. So:

• example.com has its own https://example.com/robots.txt
• blog.example.com has its own https://blog.example.com/robots.txt

If you run many sites on a single account (common for agencies and resellers), keep a small checklist for each new domain: DNS, SSL, robots.txt, sitemap.xml. Our guide on managing multiple websites on shared and reseller hosting has more operational tips that fit nicely with this.

Step 5 – Advanced Scenarios: Multilingual, Subdomains and Staging

Subdomain vs subdirectory and its impact on robots/sitemaps

Your domain architecture (blog/store/languages) directly affects how you design robots.txt and sitemap.xml. If you are still deciding, read our detailed comparison of subdomain vs subdirectory for SEO and hosting.

• Languages in subdirectories (e.g. /en/, /de/): one robots.txt and one (or multiple) sitemaps on the main domain. Sitemaps can separate languages and can be referenced from the same sitemap index.
• Languages on subdomains (e.g. en.example.com, de.example.com): each subdomain gets its own robots.txt and sitemap set.

For international SEO, make sure your sitemap structure matches your hreflang strategy, and that no language versions are accidentally disallowed.

Staging sites and test environments

Public staging environments can be accidentally indexed when indexing controls are missing, for example when:

• A staging site copies production settings without adding access restrictions or appropriate indexing controls
• A public staging subdomain has discoverable links but no effective indexing controls

On staging, protect access first; crawl restrictions are only supplementary:

• HTTP auth (username/password) or IP restriction at the web server level
• Disallow: / in robots.txt as a supplementary crawl restriction, not access protection or a guarantee against indexing. For a public staging site using noindex, allow crawling so search engines can read that directive.
• No sitemaps exposed publicly

When you clone staging to production, always double‑check that you remove the Disallow: / rule and point sitemaps to the correct domain before going live.

Handling multiple sitemaps across domains

For Google, cross-site sitemap submission has two documented methods. With Search Console, verify ownership of every included site, then submit a sitemap or index that covers those sites. Alternatively, create a separate sitemap for each site containing only that site’s URLs, host these files at a location you control, and reference the correct file from each site’s robots.txt. The robots.txt method does not use Search Console verification as its submission step. These are Google’s documented workflows; check other search engines’ requirements separately. Keeping sitemaps per domain or subdomain also makes ownership, monitoring and URL scope easier to manage.

Step 6 – Testing, Monitoring and Avoiding Silent SEO Disasters

Use Google Search Console and Bing Webmaster Tools

After setting up robots.txt and sitemap.xml:

1. Verify your domain in Google Search Console and Bing Webmaster Tools.
2. Submit your sitemap URL(s) in each tool.
3. Use the “URL Inspection” (GSC) and “Fetch as Bingbot”‑style tools to test specific pages.
4. Check for indexing coverage issues, blocked resources and unexpected noindex directives.

Test robots.txt with crawler tools

Many SEO tools can simulate how robots.txt rules apply to specific URLs. Even without those tools, you can:

• Keep a simple allow/deny matrix in a spreadsheet for critical paths
• Spot‑check with browser: request /robots.txt and confirm the live version matches your latest changes

Check your server logs for crawler behavior

Server access logs can help you analyze crawler requests that reach your server. On VPS or dedicated servers, depending on the configured log format, they can show:

• Which bot user-agent strings appear most frequently; these identify claimed bots, not verified identities
• Which paths they crawl most
• Whether recorded requests match paths blocked by the robots.txt rules in effect at the time

If you are not yet comfortable reading logs, our guide on how to read web server logs to diagnose 4xx–5xx errors on Apache and Nginx is a great starting point. The same techniques help you understand crawler patterns and detect abnormal activity.

Common silent issues to monitor

• Copied robots.txt from another project that still blocks important paths (e.g. an old Disallow: /shop/ rule kept by mistake)
• Sitemaps listing URLs that now redirect or return errors after a redesign
• Changes in site structure (new subdirectories, new hostname) without updating sitemap & robots.txt references
• Moving from www to non‑www (or vice versa) without aligning sitemaps and canonical URLs

Hosting and Infrastructure Considerations

Hosting configuration also affects how robots.txt and sitemaps are generated and served.

Performance: sitemaps on busy stores and portals

On high‑traffic e‑commerce or content sites, sitemap generation can become a noticeable load if it runs dynamically on every request. Good practices:

• Cache sitemap output in a file or object cache (Redis, for example) and refresh only when needed.
• Split gigantic sitemaps into logical pieces: products, categories, blog, static pages.
• Consider offloading heavy reports and logs; combine with a solid backup and retention strategy so your main hosting remains lean.

SSL, redirects and canonical domains

For a single site with sitemaps hosted on its canonical hostname, keep your preferred hostname (with or without www) and protocol (HTTP vs HTTPS) consistent across the following. If you deliberately host sitemaps elsewhere, use the cross-site submission workflow described above: the sitemap file’s host may differ, but the listed page URLs should still use each site’s canonical form.

• Sitemap references in robots.txt
• Sitemap URLs
• Canonical tags
• Redirect rules

For example, if you use HTTPS and no www:

• Robots.txt should say Sitemap: https://example.com/sitemap.xml
• All sitemap URLs should start with https://example.com/...
• HTTP and www.example.com should 301 redirect to the corresponding URLs on https://example.com, not send every page to the homepage

Misaligned configurations can cause duplicates, wasted crawl budget and diluted link equity.

Putting It All Together: A Practical Checklist

When setting up a new site or migrating to a VPS/dedicated server, use this short checklist:

1. Decide structure: domain, subdomain vs subdirectory, language strategy.
2. Confirm canonical URL: HTTP/HTTPS, www vs non‑www, redirect policy.
3. Generate sitemap(s): select the intended CMS core generator, plugin, build script or custom route and verify its actual index URL.
4. Serve robots.txt at the root of each domain/subdomain through the existing CMS output or an intentionally managed static file. Inspect existing rules first; do not shadow a dynamic endpoint with an empty physical file.
5. Reference your sitemap inside robots.txt with full HTTPS URLs.
6. Test live URLs: inspect /robots.txt and the advertised sitemap index, using /sitemap.xml only if that is the actual endpoint.
7. Submit sitemaps in Google Search Console and Bing Webmaster Tools.
8. Monitor logs and coverage closely for the first few weeks and after significant site changes; correct known mistakes promptly and use observed crawl patterns to guide further adjustments.

Conclusion: Small Files, Big SEO and Hosting Impact

robots.txt and sitemap.xml are easy to overlook, especially when you are busy with design, content, payments and integrations. Correct setup helps crawlers discover intended URLs and avoid unnecessary crawling. Match these files with your domain architecture, redirects and hosting configuration, and review them when those settings or your site structure change.

If you are launching a new site or planning a migration to a VPS, dedicated server or colocation, include robots.txt, sitemaps and hosting‑side SEO basics in your launch plan. Combine this guide with our new website launch checklist and our article on choosing an SEO‑friendly domain name to plan your technical checks before launch. Adapt the examples to your actual domain structure, CMS and server configuration, then test the live URLs.