Website owners and webmasters performs various actions to get their website indexed properly in search engines and they do succeed but sometimes they do not get to understand the indexing pattern or behavior and blame it on the search engine for faulty indexing, but our years of experience tells that there is always an underlying problem that can contribute to indexing issues. If you website is one of those or if you are just curious about what is going on with your website and how google often google is visiting your newly created or recently updated pages, there is a way that you can find that activity.
For reviewing the Google search bot activity, we need to setup Google Search Console account and verify our website. If this has already been done, you will need to wait for a few days to few weeks to be able to start viewing the indexing data and resolve any issues such as 404s and temporary redirects.
Once all setup, lets open Google Search Console and navigate to CRAWL STATS.
You may notice that the above image tell about the number of crawl requests in last 90 days, total downloaded size and average response time. If you have any hickups with the response time, then you may need to review your website code or hosting account resource availability to narrow down the issue and fix it. A High response time can cause lower rankings with google pushing your website away from top results, thus getting a Page Speed professional involved can help resolve such issues.
Also in the above figure, you may see that there is host status area where it may report if any issues found by Google connecting to your website.
Also, if you scroll down, you will find area wise crawl, such as “By Googlebot Type” which is our top choice for primary inspection.
From the image above, check that how Google discloses the number of request narrowed down by desktop or mobile. TO view further, lets click on the Smartphone link to open up the details of the crawl requests by google using smartphone Googlebot.
Look how google provides a detailed info on how many requests it made to each individual url, providing the details such as time and URL status, which in our case was a 200 for all the recent requests. If there are 404s and 301 and 302s, then you will need to review if there is any action required or if that action was a legit one.
This looks simple, but is powerful enough to find and fix all issues as found by Google thus allowing you to fix your website based on feedback provided by Google in its Search Console tool.
Let us know if you website needs any review and you need to fix issues on your website.
