This post is purely theoretical based on what has worked forour client, on top of this there are two scenarios:
For help with SEO or hosting then feel free to get in touch via our contact form.
This issue has happened to our client twice now, the websitefocuses on consumer tech news and reviews and has a large amount of trafficwith daily content being posted. It is therefore essential that content isindexed as quickly as possible to stay ahead of the curve when reporting newsor doing product reviews.
The website is in Google News, and content generally getsindexed within a few minutes, if not seconds.
In September, around another major Google update, the postssuffered issues with a massive delay in indexing, this could be several hoursif not longer before content shows up.
This caused issues with featured snippets and contentgetting exposure in Google News.
Furthermore, sites scraping the content would get thatcontent indexed first, which in turn would cause the client site to suffergreatly for any rankings based on their new content.
This took quite a long time to identify then fix last year,but since then traffic has grown breaking all records in April
There was a small drop in rankings during this updates, and itbecame apparent traffic was dropping off slightly with an approximate 20% drop.There are many other factors at play here, the lockdown has had a significanteffect on traffic, and the easing of the lockdown seems to correlate to a dropin traffic.
However, it quickly became apparent that content was notgetting indexed properly, and certain sites were then ranking above our clientfor the content the client had written.
This issues also affect Google news, the client has a legacy Google news listing, then they also have listed the site via Google Publisher Centre.
In the legacy listing, which we can't manage anymore,content would not get listed though you can force a content refresh with thenewer Google Publisher Centre.
We found there is a strong correlation between Google Newslisting an article and the Google AMP Cache refreshing.
While this does not match up 100% of the time, we found thatwhen the AMP cache updates showing the homepage correctly, then the Google Newsalso shows the content.
Accelerated mobile pages have been a fickle mistress; thereis strong evidence that shows AMP content has helped traffic grow to the siteexponentially and currently the AMP pages receive 44% of the websites traffic.
These pages are specifically designed for mobile devices andaim to speed up the website and offer an improved mobile experience.
Google clearly prioritises these pages when it comes tomobile search, and the website currently receives 56% of its traffic viamobile.
However, Google has two ways of serving AMP content, your actualamp pages such as (https://www.dolphinpromotions.co.uk/amp)or via their own cache content which in the case of our website is https://dolphinpromotions-co-uk.cdn.ampproject.org/c/s/dolphinpromotions.co.uk/amp
However, we have found that ampproject.org doesn't refreshtheir cache often in some scenarios; the homepage of the site is particularly problematic.
When we experience these indexing issues, the cache cansuffer from delays by days if not weeks.
Before we move onto possible solutions, we eventuallythought that it wasn't just the AMP cache that was the source of the problem,but perhaps caching in general.
The site runs on Cloudflare and uses WP Rocket. Previouslywe used Stack Path CDN.
The issues we faced also seemed to line up to CDN changes.We theorised that one possible cause was aggressive caching was meaning thatGoogle just doesn't see the content.
The first time we had this solution, we found that disablingStack Path and serving the content directly from the server improved the indexingissues considerably. This led to us ditching Stack Path totally and going backto Cloud Flare.
This time around, we also disabled WP Rocket to make surethe caching issues weren't on the server-side. From our experience, this didn'tnegatively affect the site speed, but Google Page Speed test did show a drop inthe score.
Permanently serving content directly from the server is notideal, so you will likely need to slowly re-introduce the various optimisationsmaking sure the site keeps getting indexed properly.
This is also true for WP Rocket, and we have just enabledthis.
The Ampproject.org cache should refresh automatically when auser visits the page, you can force this by vising the appropriate page for thecache, and you can find the URL via AMP.dev.
What we sometimes find works, is to refresh both the articleand homepage cache URL then also reload those pages in a separate browser or inincognito. This should simulate two users visiting the cached content forcingthe cache update.
You can tell Google to update the cache using the update-cacherequest, this is not a particularly easy solution, as it requires you togenerate your own RSA key then submitthe URL with the key.
There is a GitHub project that simplifies this process but eventhat was tricky to get working.
The easiest solution is to use the AMP for WP plugin tohandle all your AMP pages then use their AMP Cache plugin. This was updatedback in January to work natively with no need to generate RSA keys on your end.
It doesn't immediately force the homepage to show new content but it improves the speed considerably.
This is the best AMP plugin we have triedhttps://www.dolphinpromotions.co.uk/amp-cache
While not a permanent solution, if you are experiencing slowindexing issues, we recommend using Google Webmaster Tools to check if thecontent is indexed, then if it is not submitted it for indexing. We find thisnormally gets content indexed within a few minutes of submission and preventsscrapers from ranking your content.
If you find you do have a site ranking higher for you foryour own content, then a Google DMCA request can fix it. It won't stop the sitescraping your content but it works as a partial fix. Google is quick for takedowns,sometimes it is a few hours though it can take a few days .
You can only submit 10 URLs at a time, and you have tosupply an explanation per URL, so it's a bit of a chore, but it works.
While the problem appears to be fixed, content now getsindexed within a few minutes, Google news lists the sites within 15 minutes andno scraped content ranks above us. Also featured snippets appear to be performingbetter.
However, the website doesn't seem to perform quite as well as it did prior to the Core Update or the indexing issues. This was true back in September and now in May. Assuming the trend follows September, this takes a while to improve, we can only assume this is while Google builds up trust back in the site and starts to show more featured snippets and rank the site higher in Google news.
Visit our site and see all other available articles!