Sitemaps & Robots.txt in NextJS

Saroj Subedi
2 min readSep 25, 2021

--

Photo by Rock'n Roll Monkey on Unsplash

If we want to communicate and provide certain information to search engines & crawlers about your website, then it is good to have sitemaps and robots.txt file. With those on place in site, we can :

  • Define paths and location of sites which are public and private
  • Pin pages as achieve so crawler will know
  • Define priority level, page updated frequency rate
  • Provide search engines info which pages are important in order to index them faster than others

Here is basic example of sitemap:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://www.example.com/</loc>
<lastmod>2020-01-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
</urlset>

For detailed specification, check here.

Create a Sitemap:

For this example I am assuming our site is a normal blog with lots of posts.

Create a file called pages/api/sitemap.js

import { getBlogPosts } from '../utils/data'
export default async function sitemap(req, res) { const postLists = await getBlogPosts()

const entries = postLists.map(post => getSitemapEntry({
pathname: `/post/${post.slug}`
}))
entries.push(getSitemapEntry({
pathname: '/',
priority: 1
}))
const sitemap = `
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${entries.join('\n')}
</urlset>
`.trim()

res.writeHead(200, {
'Content-Type': 'application/xml'
})
return res.end(sitemap);
}
const hostname = "https://mytestblog.com"function getSitemapEntry({pathname, priority = 0.5}) {
return `
<url>
<loc>${hostname}${pathname}</loc>
</url>
`
}

Now we need to map /api/sitemap to /sitemap.xml. For this go to next.config.js file

module.exports = {
async rewrites() {
return [
{
source: '/sitemap.xml',
destination: '/api/sitemap',
},
]
},
}

We can now access sitemap in http://localhost:3000/sitemap.xml

Lets create a robots.txt file. Go to public folder and create robots.txt file and paste this:

User-agent: *
Disallow: /api
Sitemap: https://mytestblog.com/sitemap.xml

Check here if you want to have more information about robots.txt

--

--