What Is Crawl Budget?

There are millions and millions of pieces of content added to the internet everyday all around the world. Google has the job of crawling through as much content as possible, indexing them, and presenting the most suitable content to answer user queries. 

This seems like almost an impossible job. To help with this task, search engines have assigned each website a “crawl budget”. In this article we will dive in to what is crawl budget  and why you should care about your crawl budget:

What does “crawl budget” mean?

Crawl budget is the amount of pages the crawlers will go through on a website within a specific timeframe. Search engines take a look at the crawl limit (how often they can crawl without causing any issues) and the crawl demand (how often search engines want to crawl the website based on the website being updated) to calculate a website’s crawl budget.

A crawl budget is in place as search engines have a finite amount of resources. The crawl budget system allows search engines to divide their focus across the vast amount of websites on the internet. This is the way search engines prioritise their crawling efforts.

The assigned crawl budget for each website is based on:

• How much crawling the website can handle. Some website owners limit how much crawling is possible as they may encounter hosting issues if too many requests happen (crawl limit).
• Which websites are worth crawling and re-crawling based on how popular the website is and how often the website has changes made to it (crawl demand).

Crawl budget is not only based on the number of pages on a website. It is also based on other documents that exist within a website such as JavaScript, CSS and PDF files.

Why does Crawl Budget matter to you?

As a website owner, you want search engines to find and understand as many of your indexable pages as possible. It is also important that search engines do this as fast as possible. This is so you can see the benefits of any website changes/new content you have made.

Depending on how the structure of your website is set up, there is potential that crawlers can not crawl your website efficiently, leading to wasting your crawl budget. Majority of large websites have areas of the site they have as non-indexed and would not like crawlers to index, nor crawl the page. If your website structure is inefficient, there is a chance crawlers will spend time on these pages that do not matter. Due to crawlers spending time on these pages, there is potential that they do not spend time on the important pages you have spent time and effort on. If crawlers do not crawl and index your important pages, users will not be able to reach those pages through search engines.

So in short, wasting crawl budgets can directly negatively impact SEO performance. 

It should also be noted that crawl budget is rarely a concern, and is mainly on the radar of large websites, roughly 10,000+ pages in size.

If you have any concerns with your crawl budget, or would like to discuss your SEO strategy – please reach out to the Digital Carrera team!