How to optimise your crawl budget

How to optimise your crawl budget

As explained in this article, for large enterprise websites, managing your crawl budget is key to allowing your most important content to be seen by search engines, indexed and ranked so the content is served to users for relevant queries. For smaller websites (under 10,000 pages) crawl budget is less of a concern. Typically, crawlers are able to crawl through the entire website around this size.

How to find out your crawl budget?

Google Search Console’s Crawl Stats dashboard:

Google Search Console (GSC) is a free tool website owners can sign up to, which can provide insights into how Google crawls and indexes your site. Under the “coverage” section you can see the number of pages being crawled, and check any errors that could be affecting your crawl budget. The crawl stats dashboard can highlight the amount of crawl requests your website is sending. If this has declined over time you may be experiencing some issues with your crawl budget.

Accessing your log files:

Log files for your website will provide details around how search engine crawlers navigate through your site, and also provides the frequency of these crawls. By analysing your log files, you can identify any errors and issues which may be impacting your crawl budget.

Best Practice tips for optimising your crawl budget:

• Use robots.txt files to disallow crawling of less important pages:

The robots.txt file for your website informs search engine crawlers about which areas of the website it should ignore when crawling through the website. It can be used to ignore non-important pages such as admin pages; pages you would not want to rank in search engines. By correctly using and updating your robots.txt file, you can stop the crawling of certain areas of your website, forcing crawlers to focus on the important pages you’d like to rank.

• Updating your sitemap:

A sitemap is a file that lists out all the pages of your website. These do not automatically exist on your website, they need to be created; however there are tools that can help automate the updating of these files as your website grows. Sitemaps help search engines to understand the structure of your website and can help make sure they do not miss any important/new pages.

• Remove duplicate pages:

Having duplicate pages on your website causes multiple issues on your website’s performance. Duplicate pages can result in search engine crawlers having to crawl the same content multiple times, negatively impacting your crawl budget. If it’s not possible to remove duplicate content, canonical tags can be used to manage duplicate content.

• Reducing load time:

Page load speeds can also impact crawl budgets. Poor load speeds can also impact overall organic performance as page speed is a ranking factor. Slow page speeds means crawlers have to spend more time on each page to see the content. Crawlers have a designated amount of time they can spend on each page before having to move on. If the content does not load in time, crawlers may not see the content on the page, nor the links it has so further pages may not be crawled depending on the internal linking structure of your website. Addressing any page speed issues will help improve your crawl budget.

• Avoiding redirect chains and orphan pages:

Orphan pages are pages that have no internal links pointing to them, so crawlers cannot get to the content on that page. Redirect chains happen when one redirect leads in to another redirect; before the final URL is reached. Both redirects and orphan pages can confuse crawlers, wasting crawl budgets. Your crawl budget may see a positive impact if you address the internal linking structure of your website, ensuring all links are updated to the correct destination.

• Removing broken links:

Broken links will point both users and search engine crawlers to dead ends on your website. This process wastes your crawl budget. Similar to addressing the internal links on your website; addressing the broken links on your website may help optimise your crawl budget, ensuring your important pages are being crawled. 

If you have any concerns with your crawl budget, or would like to discuss your SEO plans for the year – please reach out to the Digital Carrera team!