Sitemaps & Robots.txt in NextJS

Photo by Rock'n Roll Monkey on Unsplash

If we want to communicate and provide certain information to search engines & crawlers about your website, then it is good to have sitemaps and robots.txt file. With those on place in site, we can :

• Define paths and location of sites which are public and private
• Pin pages as achieve so crawler will know
• Define priority level, page updated frequency rate
• Provide search engines info which pages are important in order to index them faster than others

Here is basic example of sitemap:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>http://www.example.com/</loc>
      <lastmod>2020-01-01</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
   </url>
</urlset>

For detailed specification, check here.

Create a Sitemap:

For this example I am assuming our site is a normal blog with lots of posts.

Create a file called pages/api/sitemap.js

import { getBlogPosts } from '../utils/data'

export default async function sitemap(req, res) {
    const postLists = await getBlogPosts()
    
    const entries = postLists.map(post => getSitemapEntry({
        pathname: `/post/${post.slug}`
    }))
    entries.push(getSitemapEntry({
        pathname: '/',
        priority: 1
    }))
    const sitemap = `
        <?xml version="1.0" encoding="UTF-8"?>
        <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
            ${entries.join('\n')}
        </urlset>      
    `.trim()

    res.writeHead(200, {
        'Content-Type': 'application/xml'
    })
    return res.end(sitemap);
}
const hostname = "https://mytestblog.com"
function getSitemapEntry({pathname, priority = 0.5}) {
    return `
        <url>
            <loc>${hostname}${pathname}</loc>
        </url>
    `
}

Now we need to map /api/sitemap to /sitemap.xml. For this go to next.config.js file

module.exports = {
    async rewrites() {
        return [
            {
                source: '/sitemap.xml',
                destination: '/api/sitemap',
            },
        ]
    },
}

We can now access sitemap in http://localhost:3000/sitemap.xml

Lets create a robots.txt file. Go to public folder and create robots.txt file and paste this:

User-agent: *
Disallow: /api
Sitemap: https://mytestblog.com/sitemap.xml

Check here if you want to have more information about robots.txt