​

Sitemap Optimization

​

8.1 The New Role of Sitemaps in the AI Era

• Traditional search engines: Discover pages, build an index, return search results
• AI agents: Discover pages, extract structured data, make product recommendations

​

8.2 Basic Sitemap Format

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/product/shoe-001</loc>
    <lastmod>2026-04-10</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://example.com/product/shoe-002</loc>
    <lastmod>2026-04-08</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

​

8.3 Sitemap Index: Sharding Strategy

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-pages.xml</loc>
    <lastmod>2026-04-10</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products-1.xml</loc>
    <lastmod>2026-04-10</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products-2.xml</loc>
    <lastmod>2026-04-09</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-categories.xml</loc>
    <lastmod>2026-04-05</lastmod>
  </sitemap>
</sitemapindex>

• Each Sitemap file can contain a maximum of 50,000 URLs or 50 MB
• Shard by content type: products, categories, blog posts, and policy pages each get their own Sitemap
• When product count exceeds 50,000, further shard alphabetically or by category

​

8.4 E-Commerce Sitemap Best Practices

​

Product Page Sitemap

1. lastmod must be accurate: Update this field whenever product price, inventory, or description changes. AI crawlers use this to determine whether re-crawling is needed
2. Tiered priority values:
   • Popular / new products: 0.9
   • Standard in-stock products: 0.7-0.8
   • Delisted products with retained pages: 0.3
3. Only include accessible URLs: Do not include 404 pages or redirect targets in the Sitemap
4. Canonicalize URLs: Each product should have exactly one canonical URL (avoid duplicate URLs with query parameters)

​

Category Page Sitemap

<url>
  <loc>https://example.com/category/running-shoes</loc>
  <lastmod>2026-04-10</lastmod>
  <changefreq>daily</changefreq>
  <priority>0.7</priority>
</url>

​

Policy Pages

<url>
  <loc>https://example.com/privacy-policy</loc>
  <lastmod>2026-03-01</lastmod>
  <changefreq>monthly</changefreq>
  <priority>0.3</priority>
</url>

​

8.5 Dynamic Sitemap Generation

​

Principles

1. Reflect database state in real time: Automatically update the Sitemap when products are listed, delisted, or repriced
2. Incremental lastmod updates: Only update entries that actually changed
3. Cache with periodic refresh: Sitemaps can be cached (e.g., 4 hours), but proactively refresh when products change

​

Platform Solutions

​

8.6 Notifying Search Engines and AI Agents of Updates

​

robots.txt Declaration

Sitemap: https://yourdomain.com/sitemap.xml

​

Ping Search Engines

# Google
curl "https://www.google.com/ping?sitemap=https://yourdomain.com/sitemap.xml"

# Bing (IndexNow is more efficient)
curl -X POST "https://api.indexnow.org/IndexNow" \
  -H "Content-Type: application/json" \
  -d '{"host":"yourdomain.com","key":"your-indexnow-key","urlList":["https://yourdomain.com/new-product"]}'

​

IndexNow Protocol

1. Generate an API key
2. Place the key verification file in the root directory
3. POST to the IndexNow API whenever a page is updated

​

8.7 Common Mistakes

​

8.8 Validation Tools

1. Google Search Console — Sitemaps section: submit your Sitemap URL
2. XML Sitemap Validator — Validate format
3. Manual check: Visit yourdomain.com/sitemap.xml and confirm the format is correct and URLs are accessible

​

8.9 Self-Check Checklist

• Sitemap exists and includes all in-stock product pages
• lastmod reflects actual update times
• No 404 or redirect URLs are included
• Sitemap location is declared in robots.txt
• Submitted to Google Search Console
• Sharded if product count exceeds 50,000