Crawl budget refers to the amount of resource (time/kb) which Google assigns to your page to crawl every day. Also, it is the amount of pages Google indexes and crawls websites within a specific time frame.
Crawl budgets to help with SEO What is the reason why a crawl budget is crucial for SEO?
In simple terms: If Google doesn’t find a website and doesn’t index it, it’s unlikely to be ranked in any way. If the number of pages is greater than the crawl budget of your website, then you’ll have pages on your website which aren’t indexable.
As you can see, this is an essential aspect, particularly on large pages (more than 1,000 pages). For smaller pages, it’s not recommended to be obsessed with it as there will be no differences.
Calculating the crawl budget: What is the process by which Google calculates/assigns the crawl budget?
The crawl budget Google assigns to your site is primarily determined by:
- Page Authority
- Frequency of publication or updates (you can have the exact quantity of URLs, but you need to keep them current and increase the crawl rate)
- Time to load the web
That is, the more links your site receives and the more material you post, and the shorter the time required to load your site, The more crawl budget will be assigned to you by Google the robots of its.
Are we optimizing the crawl budget? How can we improve our crawl’s funding?
Let’s look at the possibilities (beyond that obvious one) we have to improve the crawl rate of our website budget:
- Take down links to pages that have errors or redirects
- Optimize the load time of web pages
- Enhance the speed of updates and publication
- Enhance the number of backlinks that link to your site
This is the start of the “beautiful” portion… This is extremely crucial to stop Google from crawling websites that aren’t relevant to placing (duplicate content, pages with a low value for SEO, etc.). To do this, I suggest:
- Create a noindex value for the page
- Remove the sitemap page (if it is automatically generated)
- Block this webpage from the robots.txt:
- User-agent: *
- Disallow: /page/
- Obfuscate links
Are you in need of help optimizing your crawl’s budget?
Contact us, and we’ll provide you with a personalized solution.
How do we hide links to maximize our budget for crawling?
To disguise a link, we will need to include Javascript code. The code that is applied, illustrated by an example, will be similar to the following:
Legal Notice
But, there’s a WordPress plugin designed by SEO professional Fede Gomez that I use to block hyperlinks on any WordPress website without having to include one sentence of Javascript or CSS code. It is available for download by clicking here. Click on the ‘DESCARGAR.’
This plugin lets you block links in 3 areas that comprise WordPress:
- Menu elements.
- Links in the post’s content and pages.
- Links are embedded within the text of HTML widgets.
To hide a link, you must implement the ‘ljoptimizer’ class, and the plugin will perform the job with PHP and Javascript functions. There is more information about the use of the plugin in this post. (sorry, this post can only be found in Spanish. However, I do hope that you can make use of Google to help translate!)