Understanding Google Updates: Analyzing Impact and Avoiding False Conclusions
Any apparent or confirmed Google Update tends to create interest, excitement and, usually, confusion amongst search marketers. Often it will be clear that an update has taken place, as fluctuation in the SERPs is greater than usual or whole new features, like video carousels, appear and wreak havoc with the rankings. The following is an attempt to filter out the background SERP noise in an attempt to see which signals can be associated with Google Updates. This should help determine which domains have most likely been targeted by the latest algorithm changes.
Have you been affected by a Google Update? Then request a website audit today.
Request an auditThe problem with winners and losers
Once a Google Update lands, the obvious first port of call for any analysis is the latest lists of winners and losers. Any domains affected by the update should experience a larger than usual increase or decrease in their SEO Visibility, and can therefore be expected to appear in these lists. This theory seems to be backed up when we look at the lists and see a large number of domains in both the winner and loser lists from the same industry, such as health. In such an instance, it’s a logical conclusion that rankings for health keywords have probably been affected by the update, as was apparent from Google’s E-A-T Update in August 2018.
Getting distracted by the giants
This approach isn’t wrong, by any means. But one thing it ignores is any underlying information about the winning and losing websites. If we look at absolute changes in SEO Visibility, then more visible websites will automatically appear more often in these lists, whether there has been a Google Update or not. For Wikipedia, with a base SEO Visibility of over 50,000,000, even a 1% change means 500,000 in absolute terms — which exceeds the total SEO Visibility of all but the top 120 websites in Google’s US index.
Percentage changes
An alternative to absolute changes is to look at percentage changes in SEO Visibility. However, in this case, smaller websites can be overly represented, as only a couple of new rankings can have a disproportionately large impact.
Another issue with percentage changes is that the SEO Visibility of some websites fluctuates more from week to week than others. One example is youtube.com. The volatility of results in video carousels, in particular – and the fact that the content on youtube.com is constantly updated – mean that rankings go up and down, and appear or disappear from week to week more often than on a site with more static content.
All the biggest weekly changes belong to YouTube. Wikipedia’s biggest change is +1.7%, whereas YouTube has several weeks with a change of greater than 2% (up or down).
Measuring expected fluctuation
If both absolute and percentage changes are potentially problematic, is there another way of measuring visibility change that allows us to compare across domains, regardless of their size, number of rankings and content volatility? The answer is: let’s try to find out.
The aim is to find out the expected fluctuation of any given domain. To do this, we can see how its SEO Visibility has changed each week (e.g. over the past five years), available as absolute values and percentages. If we take this as our background data set, we can calculate summary statistics, such as means and standard deviations that give us domain-specific insights into how volatile a website is.
| Statistic | youtube.com | wikipedia.org |
| Average change in SEO Visibility | +86,346 | -26,172 |
| Size of average change | +541,025 | ±323,702 |
| Largest decrease | -6,069,941 | -3,578,060 |
| Largest increase | +9,054,350 | +3,105,531 |
| Standard deviation | 1,125,894 | 526,253 |
| Average % change | +0.33% | -0.04% |
| Size of average % change | ±2.02% | ±0.59% |
| Largest decrease % | -25.1% | -6.1% |
| Largest increase % | +25.6% | +5,6% |
Introducing the z-score
Using these statistics, we can look at the change happening for a domain in one week in the context of how this domain is normally expected to change. A metric called the z-score expresses how many standard deviations from the mean a value is. As this measure is effectively standardized for both domain size and volatility, it ensures comparability across domains.
Using the z-score of a domain’s change in SEO Visibility is a more reliable measure for establishing whether that week’s change was significantly unusual. As a rule of thumb, any z-score over +2.0 or under -2.0 suggests a significant divergence from the mean.
At first glance, the z-score chart might not look all that different from the percentage chart, with the two biggest drops coming for YouTube in weeks 14 and 22. However, on closer inspection, the z-scores tell quite a different story. Look, for example, at week 28. On raw percentages (above), YouTube is well ahead of Wikipedia, but the z-scores show us that this was a quite unusual week for Wikipedia, with a z-score of 1.41, whilst it was nothing out-of-the-ordinary for YouTube, which recorded a score of just 0.75.
Turning theory into practice: March 2019 Google Core Algorithm Update
To see how this theory might be applied to a Google Update, we can look at data from the Google Core Algorithm Update in March 2019. The chart in the first tab shows a selection of the top winning domains, sorted according to their percentage increase in SEO Visibility. The second chart shows the z-scores (calculated on the basis of 5-year SEO Visibility history).
It is immediately obvious from the two charts that the percentage fluctuations do not go hand-in-hand with the z-scores. The domains that are of most interest (with the largest z-scores) are:
- patriots.com (5.3)
- allmenus.com (5.2)
- 247mahjong.com (5.1)
- wellsfargoadvisors.com (4.6).
This does not necessarily mean that all of these have seen such radical change just because of the Google Update —any z-score over 5 is so high it is likely something unrelated (also) happened — but it does mean that these domains have experienced the most unusual behavior this week, more so than gossipcop.com or freeonlinegames.com. So it’s these domains that should probably be looked at first, if trying to understand the impact of the Google Update.
So now we can see definitively which the domains were most affected by a Google Update?
Unfortunately not. Even with a metric that improves comparability of domains, a great deal of caution is required when analyzing the impact of a Google Update. The z-score doesn’t explicitly determine whether there has been a Google Update or not — it simply indicates that the rise or fall in SEO Visibility has been unusually large. There are several other common reasons for large SEO Visibility changes, that have nothing to do with Google Updates:
- Relaunch: Following a relaunch, a website may have new content and a completely new internal linking structure. This can have a major effect on how Google evaluates the site.
- Migration: When a website is migrated to a new domain, Google will have to re-index all URLs, which can cause problems. For example, issues with broken redirects can impact the crawlability of the site and/or impact negatively on the website’s backlink profile.
- New content: A domain may launch a whole new section or category page, such as a magazine. A major alteration to the content on the domain, once picked up by Google’s crawler, is likely to impact rankings.
- Anomalous keyword spike: Particularly for smaller websites, one or two erratic keyword rankings can have a big impact on SEO Visibility. If a domain suddenly has a top-10-ranking for a high-volume keyword, this can skew the SEO Visibility in a big way. For example, searchmetrics.com once started ranking at position 2 for “call mum”.
If you have had the ill-fortune to relaunch or migrate your website around the same time as a Google Update, and you have suffered a major drop in SEO Visibility, then this is where you should start. An investigation of possible errors made in the relaunch process is highly recommended, before you try to analyze how the algorithm change may have affected your rankings.
Common pitfalls
Having attempted to provide a more considered view of Google Updates, here are a few further traps to avoid falling into when you log in to the Searchmetrics Suite on a Monday morning and see what looks like a huge change in your SEO Visibility.
An SEO Visibility drop doesn’t always mean a traffic drop
SEO Visibility is Searchmetrics’ universal metric that makes it possible to compare domains and how well they are represented on search engine results pages. If you do see a big drop in your SEO Visibility, then you want to find out why and you want to do something about it. However, while SEO Visibility is a useful indicator, it doesn’t always tell the whole story – you should also look closely at your website analytics report, particularly the traffic coming to your most important landing pages, which may not have changed in the same way.
Alterations in the SERP layout, or anomalous keyword rankings like that mentioned above can impact your traffic in ways that won’t always be reflected in your SEO Visibility. That said, if you do see changes in traffic, then the SEO Visibility is often a great starting point for your investigation into the causes. The relationship between SEO Visibility and traffic is described in more detail in the Searchmetrics glossary.
SEO Visibility FAQs
Are you bouncing on the SEO Visibility yo-yo?
The obvious way of measuring a Google Update’s impact is to look at changes in SEO Visibility during the relevant week. However, it is quite common for websites to experience a spike the week after a drop, and vice versa: to first shoot up and then drop down. A simple reason for this could be a few unusual rankings that changed for a short period, without there being anything fundamentally different about the website or Google’s assessment of it.
The z-score approach goes some way to countering the impact of this kind of volatility, but it is nevertheless worthwhile, if a website does have an unusually high jump or low drop, to see if there is a yo-yo effect the week after. If not (and if still not another week later) then the change is likely to be more than a blip, and it makes sense to start drawing conclusions and looking into the root causes.
The chart shows an example of an SEO Visibility yo-yo around the March 2019 Core Update. cargurus.com goes up and drops down, while yellowpages.com drops and bounces back up.
Weekly percentage changes need context
If reporting changes in SEO Visibility, percentages are most useful for middle- and long-term trends, but potentially misleading if applied to a one-off week. This is particularly true when the SEO Visibility yo-yo is at work. If you say, “increased by 25% this week”, but there was a drop of 20% the previous week, then that is zero change over the two-week period. Alternatively, if you have an 20% followed by a 18% increase, then the second week has seen a larger absolute jump. Percentage changes can be used to indicate the impact of a Google Update (or other major event) but, as with anything, they need context to be understood properly.
Don’t ignore seasonal winds of change
Changes in SEO Visibility should not always be unexpected. While some websites will have content that is relevant all year round, others will have peak times of user interest. Examples include shops that offer products with an obvious seasonal focus, such as winter sports equipment. The following chart shows the SEO Visibility of fifa.com. With no context, the 50% rise in the summer of 2018 might suggest some major change in Google’s algorithm.
If, however, you know that the Fifa World Cup took place from 14 June to 15 July 2018, then that puts the Visibility jump in perspective. This is supported by a look at the monthly search volume for the keyword “fifa world cup”.
This is an extreme example, but one that demonstrates how an increase in SEO Visibility could mean that your content strategy is working, if you are targeting a certain time of year. Just as many businesses will have different sales targets for different quarters, website traffic should not be expected to remain constant. As in the FIFA example, a subsequent decline doesn’t mean that anything went wrong, it should be an expected part of the website planning. Ideally, it would make sense to develop a strategy designed to maximize the sustainability of seasonal peaks — temporary increases in user interest are a good opportunity to expand the regular user base, and can provide stimulus for longer-term improvements.
Watch out for video volatility
YouTube’s erratic ups and downs were discussed earlier. The impact of video search results with a short half-life doesn’t only affect this one website. Other domains that get a significant proportion of their search visibility from video content are also likely to be vulnerable to similar fluctuations. A comparison of the video and overall SEO Visibility of espn.com shows that viewing one without considering the other can mean ignoring a significant part of the picture.
Of course, not all videos are news-related or contain short-lived viral content. Depending on the topic, video can also be used for evergreen subjects, such as how-to guides or in-depth explainers. If a high-traffic piece of video content that has been a mainstay at the top of the SERPs suddenly vanishes – and stays out of sight – then this can be just as serious as the loss of any other organic rankings.
Google Update analysis on the Searchmetrics blog
The kind of analysis described here requires time, both to fully establish and understand what impact the update has had on your domain and to go in-depth into possible causes. This kind of analysis is recommended before making any major decisions regarding how to respond to an update. By contrast, the analysis reports published on the Searchmetrics blog, such as this one on the June 2019 Core Update, provide an overview of the background to the update, and try to identify more general trends affecting the SERPs as a whole, and the possible direction Google’s algorithm changes may be taken. The analyses on the blog should not influence any major decisions before you have more closely examined the specific characteristics of your domain.
Conclusion: For Google Updates, context is king
The common thread throughout this discussion of Google Updates and fluctuations in SEO Visibility is the importance of context. If your performance drops, you need context that tells you things like:
- whether this drop is unusually large for your website, or is just noise,
- whether competitor websites have seen similar or opposite drops,
- which other factors, besides a Google Update, could be relevant,
- which keywords (or types of ranking) have been affected.
Exploring these details gives you the information you need to understand the changes in your performance, and where they might be coming from. If this means that context is king, then data is the kingmaker. Without historical data about your (and your competitors’) domains, without broad, deep keyword ranking data and without URL performance data, you’ll be lacking the context you need to draw accurate conclusions and to avoid erroneous assumptions. This is essential if you hope to make changes that will address real weaknesses and positively impact your website performance for long-term success.
