Unwanted Resources in Disallow in robotstxt
Posted: Mon Jan 27, 2025 5:19 am
Robots directives are used to block the crawling of paths that we do not want Googlebot to follow and to avoid the consequent waste of crawling resources intended for our web project ( crawl budget ). Now, knowing this, you cannot simply take all the irrelevant paths and throw them into the robots file lightheartedly. Let me explain: if half of the explicit URLs in the source code of ALL the pages of the website refer to paths to be blocked, we force Googlebot to make distinctions for each page that we give it to crawl. We are basically asking for a huge effort that could be avoided simply by serving the same loads as asynchronous paths, i.e. without href attribute.
The big difference between synchronous and asynchronous paths is in the scanning the usage of facebook database priority. The former are followed almost always, the latter only if the bot deems it necessary. Understand?
To learn more, you can read: Crawl Budget: Here's Why You Need to Optimize It .
This type of SEO requires the intervention of developers, coders, developers or whatever the hell you want to call them. You can't do without these technical figures, just as web marketing can't do without us SEOs.
After all, when an e-commerce site with 1,000 products including bags, backpacks, suitcases and accessories, finds itself with 100,000 scanned pages in index coverage on Search Console, who do you think you're going to call, the Instagram consultant?
The big difference between synchronous and asynchronous paths is in the scanning the usage of facebook database priority. The former are followed almost always, the latter only if the bot deems it necessary. Understand?
To learn more, you can read: Crawl Budget: Here's Why You Need to Optimize It .
This type of SEO requires the intervention of developers, coders, developers or whatever the hell you want to call them. You can't do without these technical figures, just as web marketing can't do without us SEOs.
After all, when an e-commerce site with 1,000 products including bags, backpacks, suitcases and accessories, finds itself with 100,000 scanned pages in index coverage on Search Console, who do you think you're going to call, the Instagram consultant?