Making your website as easily crawlable for search engines/search bots seems logical right? But when you take a closer look at a vast number of sites, they’re losing out in terms of allowing the search engines to be able to crawl, read and understand a site. Crawl budget is REALLY important for SEO, so having a site that is as error free as possible and easily crawled needs to be included in any SEO strategy.

We will discuss what is crawl budget, how you can improve/optimise for it and some pointers on what needs to be reviewed on a regular basis.

Optimising Crawl Budget - Digital Thrive

So What is Crawl Budget?

When your website gets crawled by a search engine crawler or search bot, Crawl Budget is simply the frequency of how many times that bot hits your site and crawls the pages of your site. That frequency can change, which is all dependant on how easy it is for the bot to crawl your site, the limiting of the amount of dead ends and errors on site. Ideally, what we would like to happen is that a search bot crawls the site more often, searching for new content, pages and updates – which inturn it can update its search results – when users are looking for information, products and services via its search engine.

Crawl Budget optimisation is focused on amending areas of your site, to make it as quick as possible for the bots to find everything it needs and increase the rate in which search engine bots visit your website and pages. The more often is visits, the quicker any onsite changes can be added to its index, which should yield better ranking results for your site – sounds logical yes! Unfortunately, Crawl Budget optimisation does get overlooked for lots of other areas when it comes to SEO, but hopefully the following steps (and if you keep an eye on them and update regularly) then it should help with making sure your site gets crawled, indexed easily and quickly.

Crawl Budget is Often Overlooked for SEO

There is a great blog post written and published by Google back in 2017, that discusses Google view on Crawl Budget – it does state that ease of crawling is not a factor that gets taken into consideration when it comes to ranking.

So when it comes to SEO’s working on sites and so on, then crawl budget optimisation becomes seemingly less important than other areas which they know they can improve, which inturn, would increase the volume of organic traffic. When building out a robust SEO Strategy for any client, I’ll factor in as many areas as possible to improve the site – whether its “deemed a ranking factor” or not, as I believe we need to look at the wider picture for clients organic traffic performance as a whole.

Now – here is a small caveat – if your site is relatively small, and I mean under 1000 pages (its not a golden rule/number), but then Crawl Budget isn’t going to be too much of an issue – yes, you want any new content, or amends/changes indexed as quick as possible (which you can do through Google Search Console’s URL inspection and if your sitemap.xml is upto date then, force that to refresh too), but its not as though you have 100k+ web pages and most of them would be old, outdated content. We would factor this in when working on a client SEO Strategy, but with a smaller site, there would be more gains in other areas.

If you do have a large site, then managing the crawl budget makes a lot of sense, simply because there will be a number of pages that you update frequently, or add that you want to be indexed and inform search engines of the changes.

We have heard a number of times over the years from main players at Google that Crawl Budget really only applies to sites with millions of pages – BUT – if you can apply some small incremental, marginal gains to a site, here and there as a part of overall improvements, then they can quickly add up to overall successes. So, yes, I am ignoring some advice from Google main players, and yes, I will factor in Crawl Budget Optimisation when it comes to clients SEO Strategies. Plus, on top of all this, when we work with larger sites – then handling any errors that come from SEO Technical Audits are easier to find, minimise and amend, which improves the health of site for search engines.

Now on to some of the tips for optimising for crawl budget, making sure there is nothing on your site that is adversely affecting how, when and what a search bot does when it comes to crawling your site.

Optimise Your Crawl Budget

When it comes to the “what can I do” or “do I need an SEO/Dev to do” then there will always be parts of your site that you can change, implement and do – there could also be other parts, which you may need someone to help you with, if its more technical than you understand… so here goes.

Robots.txt File – Make Sure Your Important Pages Can Be Crawled

The robots.txt file is a simple txt file that sits at the root of your website – e.g. – it allows you to control what part of the sites that you want/don’t want a search bot to crawl – you can use it to block admin parts of your site, cart/basket etc etc. Its one of the easier steps/additions to any site – Google has a great introduction to robots.txt file to read.

There are lots of ways to manage the robots.txt file, if its a small site – then I will usually do this manually by hand in Notepad. If its a much larger site, then I will use an auditor tool, as they are quicker than manually amending it. Within the robots.txt file we will instruct search bots which pages to ignore and which pages we will allow them to crawl – this optimises your crawl budget, by telling them to spend their time on pages that you want crawled and those that are either not important, or as above account, admin parts, cart, basket type pages. If you are working on a large site, that has a forum for example – which is closed to non subscribers, then you may want to block the whole forum, which inturn search bots will not spend time crawling and not index, spending time on more important pages on your site.

You can upload the robots.txt file to the root of your domain and its done.

Redirect Chains are a Drain!!

I rarely come across a website when running a Technical SEO Audit that doesn’t have redirect chains in place – they are so common, especially if a domain/website has a lot of history and been around for a long time. Redirect chains are multiple 301 redirects, where a page/URL used to live, which has then been redirected to a new URL, which inturn gets redirected again and again – this is a redirect chain. I have seen some bad examples of 25+ redirects working across some sites – it may not seem so bad, as at least its not a 404 Error, page does not exist – but as mentioned earlier, search bots want to crawl and understand your site as quick as possible. If you make a search bot jump through lots and lots of chained redirects, then its wasting its time and “budget” on those, rather than on important, fresh updates or content.

Making sure that you are on top of any 301 Redirects is key to a website health overall – minimising them as much as possible is a good approach. If you’re unsure if you do have any, or looking to find where they are and how to fix them – then a Technical SEO Audit should be done, where you should be supplied with where they are and how to fix them. Having a few on your site isn’t going to be the worst thing, but “house keeping” to limit them is key for crawl budget optimisation.

Fix any HTTP Errors

As with the Redirect Chains above, any 404 or 410 Errors on your site use up the crawl budget, because search bots spend time trying to locate the URLs (usually the URLs have been hyperlinked to from content on your site, or hyperlinked by external sites), when they don’t find them, then that crawl has been a waste of time.

Making sure your site is free from any 4xx errors is key to the sites overall health too – so finding them and fixing them is a must for me. We use a number of SEO Auditing Software platforms, but my go to is Screaming Frog – its free to download and use upto crawling 500 URLs on your site, so if your site is smaller than this, then makes sense to use it.

The All Important Sitemap.xml File

As with the importance of the robots.txt file – you need to make sure your sitemap.xml is both upto date and includes the URLs across your site. The robots.txt file and sitemap.xml file are the two files that search bots can and will quickly access to understand what and where to crawl, the URLs, folder structure and information architecture of your site.

Search bots crawling your sitemap, will quickly know and understand the context of your internal links, where they lead, what folders they are in and so on. Make sure that the URLs within the sitemap are upto date, with how they are accessed and viewed on your website. Also include the URL of the sitemap within your robots.txt file, e.g. Sitemap: – its as easy as that.

The sitemap.xml files also “lives” at the root of your domain – many website platforms autogenerate (aka WordPress) them these days, which is kind of ok, but I prefer manually updating them, or extracting them from Screaming Frog and uploading myself, once I am happy with them.

The Old HTML v Javascript Rendering/Understanding

Age old debate that rattles around year in, year out – where Google states “We can and do render and understand Javascript” when it comes to understanding the context and content of your site…. erm, yeah ok, we don’t believe them and would rather that sites are built in HTML and rendered server-side and not client side (most sites are rendered server-side, so don’t worry, if you are unsure).

When it comes to a search engine understanding the content, relevancy of your website and its pages – then, our stance is to make it as easy as possible for them to work out what your site is about and build it in HTML. This whole article is about the speed and ease of search bots to crawl your site and optimising crawl budget, if your content is built using Javascript, which needs to be “pulled in and crawled” in an external file, then that doesn’t sound optimal, or easy for a fussy search bot to do.

In Conclusion

For us, we include crawl budget optimisation into client SEO Strategies – every percent or marginal gain you can garner and implement can quickly add up when competing for top spots and increasing organic traffic to your site. If you’re a brand/site owner, with a sizeable website and discuss this with your SEO Agency or freelancer and they shake their heads, shrug their shoulders, then ask them to at minimum to undertake the above. Most of the above would come out in an SEO Technical Audit, so if they haven’t recommended or undertaken one, then just ask them to.

If you are looking for help with your SEO Strategy, a SEO Technical Audit or would like a discussion about your site, then please get in contact today.

Get in Touch