Noise or Music? - The Insights Blog

“not provided” – Organic search terms blocked by Google

October 19, 2011 / Categories: Metrics understanding, Privacy and Accuracy / Comments: 26

Share Button

An odd announcement form the GA product team was made last night that affects all users of web analytics tools:

When a signed in user visits your site from an organic Google search, Google Analytics will no longer report the query terms that the user searched on to reach your site.Full announcement

That’s a BIG change! Essentially marketers will no longer be able to view the keywords used by visitors that come from Google organic search in their web analytics reports (the fact they came from a Google organic search is still shown). Instead the keyword will be listed as “not provided” The rational for this, as the announcement explains, is to protect the privacy of users who are logged into their Google account.

What’s my view?

As you know, I am a big fan of user privacy and putting the end-user experience first, so on the one hand this is a good approach. However, this does appear to be a little over zealous.

Oddly, paid search is treated differently…

A dichotomy form the announcement is that Adwords traffic is not affected by this. It appears strange to me that Google considers privacy important for organic searches, but not for paid searches.

What’s the impact?

This impacts Google Analytics users and users of *any* web analytics tool. Only Keyword=(not provided) is shown.

And I suggest the impact is very large. Although users who are not logged into Google are unaffected, there are approximately 300m GMail users globally, and anyone who is active with Google services (Google’s whole raison d’etre!) will fall under this catch-all setting. It means a valuable piece of information for optimising your website – both for content and traffic acquisition – is going to be missing form your reports. I find that peculiar coming from a company that focuses heavily on keyword relevance when ranking websites in its organic results. Surely, should be helping website owners with this info…

The following video illustrates the significant impact this is starting to have for the period 10-Oct 2011 – 08-Dec 2011 (idea taken from Michael Whitaker‘s similar observations):

Click to play video showing the impact of 'not provided' google keywords

Your thoughts

This is going to be a major talking point I am sure and I would love to hear your feedback. The comments on the announcement itself are very negative (I particularly like the first one – “where’s the -1” – joking a side, I do think a -1 button would be great to have in search results).

For me, I really do not think this has been thought through fully by Google (something I have noticed happening more often at G in general). I agree the user should have the option to keep their search terms private – the option to switch this on/off in the account settings would make sense. However, at present keywords are removed whenever a visitor is logged in to their Google account – with no choice for the user (I don’t consider logging out a viable option). Thats got to be a bad thing, no?

Share Button

Comments

  1. Update:
    The latest Firefox, version 14, now automatically uses encrypted search for Google. That means all Firefox 14 (and upwards) visitors with show as “not provided” for Google organic searches.

    Thanks to Dara Fitzgerald of FreshEgg for bringing this to my attention

  2. Tony Lopes says:

    My view is simply that Google wants to make more money from Google AdWords and by making SEO more difficult marketers will tend to spend more on paid search. The argument that it is for privacy protection holds no water if the same protection is not applied to paid search.

  3. Some numbers on the impact of this from Search Engine Land (via Daniel Waisberg):
    http://searchengineland.com/encrypted-search-terms-hit-google-analytics-99685

  4. Dimitris says:

    Its infair to the SEO industry… Its like a war between Google ads and Seo…. am i right?

  5. Alan Perkins says:

    @Brian, I have gone into this in more depth on my blog at http://bit.ly/oVISbY and, by the look of it, have conducted some similar experiments to Birger and yourself. Those experiments show that the search results URL might be something like:

    https://www.google.com/#hl=en&sugexp=kjrmc&cp=2&gs_id=6&xhr=t&q=car+insurance&tok=NXk1jCeRGLusJVKoMU0HeQ&pf=p&sclient=psy-ab&site=&source=hp&pbx=1&oq=ca&aq=0p&aqi=p-p2g2&aql=f&gs_sm=&gs_upl=&bav=on.2,or.r_gc.r_pw.r_cp.,cf.osb&fp=ec7fae886d32900d&biw=1680&bih=913

    You’ll see that this contains the query “car insurance”, plus a number of other query parameters which may help to identify me personally [should I be worried? :D].

    When I click on a natural result, and then look at my referrer, I see it is now:

    http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&sqi=2&ved=0CHUQFjAA&url=http%3A%2F%2Fwww.moneysupermarket.com%2Fcar-insurance%2F&ei=AKWmTtb0FcmYhQeNuOD7DQ&usg=AFQjCNF-UvvfJsjMbuyeGwVVnyzkQmInRA&sig2=EBM-hDKqw-m6VT55J3WOzw

    The query (&q=) has been stripped away, and in fact the URL is completely different to the actual referring page. My point is, if Google is in such complete control of the referrer, why did they keep all those query parameters I don’t need (sa,rct, esrc, ved, usg, etc(!)) and remove q? I would be quite happy with

    http://www.google.com/url?q=car+insurance

    and I don’t see how this violates privacy. Bear in mind the reason given in Google’s original blog post was “As search becomes an increasingly customized experience, we recognize the growing importance of protecting the personalized search results we deliver”. So why not de-personalise the referrer, rather than remove the keywords?

  6. @Brian in comment 20. I completely agree that the WMT data is useless at addressing the post-click behaviour analysis that’s been so useful in the past. By the same token as Matt Cutts’ comment, I suspect there are many that would say that the data in GA is different from that found in the server logs!

    This won’t help you get the data but you can gauge how concerned you need to be by following this post from Avinash: https://plus.google.com/105279625231358353479/posts/iWYvxFMMZH9

    Since my clients are in Europe we’ve yet to see significant impact yet but I’m sure it’ll come.

  7. @James Docherty
    There is Webmaster tools data in GA, but I find that superficial as a substitute. In WMT, you get the top 1000 terms over the past 30 days, but that’s it. What happens next i.e. time on site, page depth, conversion rate, engagement, transactions – in fact all of the things we, as analysts, have been looking at for the past decade have disappeared at the keyword level.

    This quote from Matt Cutts is pertinent:

    Please don’t make the argument that the data in our webmaster console is equivalent to the data that websites can currently find in their server logs, because that’s not the case.http://news.ycombinator.com/item?id=2122786

    @Alan Perkins
    From a search point of view, the potential for personalised info is in the query parameter. Other URLs – such as Facebook, may contain personal info in the page path part of the URL. So, this is not the same thing. Or do you mean something else?

  8. Alan Perkins says:

    Brian, as you’ve pointed out, Google is completely in control of what data they leave in the referrer URL and what data they take out.

    So, why do you think they left most things in the referrer and took the keywords out, rather than leaving the keywords in and taking everything else out? i.e. de-personalise the URL, if personalisation is the issue …

  9. It strikes me this is an attempt to break all those 3rd party services/tools that either compete with GA or report massively misleading data.

    It looks like the search query data for your site is still going to be available in Webmaster Tools.

  10. @Brian

    I completely agree with you and that’s what all my tests revealed as well. Google Encrypted Search doesn’t modify the referrer at all it’s more how browsers handle the site change from https to http.

    Don’t forget that not all advertisers use Google Analytics for AdWords campaign tracking. :)
    Third party tools still analyze referrer data to grab the real search term from the referrer in order to allow keyword/search term analysis. This wouldn’t be possible when Google also removed the ‘q’-parameter value from PPC clicks as well. But they haven’t done it (yet). That’s what I tried to analyse in my 2nd update of the article (table):

    http://www.dc-storm.de/news/Google+k%C3%BCndigt+SSL-Suche+an+-+Gedanken+dazu/#update2

    No matter if it’s http://www.google.com or https://www.google.com Google doesn’t remove the ‘q’ parameter value (&q=iphone in this example) from the PPC referrer.

    So it’s 100% under Google’s control what they want to show and what they want to suppress.

  11. @ Birger

    The general rule:
    Browsers do not transmit referrer information when navigating from a https page to an http page.

    If you click a link in the search results at https://encrypted.google.com, your browser navigates directly to the site. If the site is on http your browser does not pass the referrer information so to ga.js it appears to be a ‘direct’ navigation. However, if the site you click on is https, your browser will pass the referrer information and ga.js will detect a google/organic search *including* keywords.

    For https://www.google.com, which will now be automatically used for logged in users, the behaviour is different. When you click on a result page your request is first redirected through an http hosted redirect page. As a result the referrer is always provided to the site, but this new referrer only indicates that a Google organic search occurred and does not include the actual keywords. This is the case that results in a google/organic campaign with the keywords set to “(not provided)”.

    Since ga.js detects paid clicks by the presence of the “gclid” parameter in the URL, paid click detection is not affected by SSL search. This is also true for any campaigns specified using the “utm_*” URL parameters. I am at a loss as to why this has not been consider as ‘private’ by G as well.

    Like I say, it feel like it been badly thought through rather than any conspiracy for people to upgrade to GA premium or spend more on Adwords…

  12. @DavidS

    This “feature” is being rolled out for google.com this and next week. Expect the other Google domains to follow suit shortly after.

  13. @DavidS I just tried the following:
    https://www.google.com/webhp?hl=en&tab=ww#hl=en&q=iphone

    Clicked on the
    Apple – iPhone 4S – The most amazing iPhone yet.
    http://www.apple.com/iphone/

    listing and tracked the referrer with Live HTTP Headers.

    This is the result that I got:
    Referer: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&ved=0CFEQFjAD&url=http%3A%2F%2Fwww.apple.com%2Fiphone%2F&ei=fCGhTvarJsbvsgbV_OHqAg&usg=AFQjCNEVhNcLPP21m2nVINX2GhlA6azJWg

    Instead of “&q=iphone” the referrer contains only “&q=”. You see what I mean?

  14. DavidS says:

    I’m not able to verify that what Birger wrote (and what Google announced) is actually taking place.

    I continue to see the q param and value exposed in the referrer on the landing page both while logged out of Google and logged into Google…

  15. Luca says:

    @Birger, this was a great post.
    When ‘q’ parameter value from the referrer gets erased, there is no chance that any web analytics tool will be able to pick up the query terms.

  16. Brian, I agree with you.

    I’ve made some extensive referrer tests and posted them on our (German) site:
    http://www.dc-storm.de/news/Google+k%C3%BCndigt+SSL-Suche+an+-+Gedanken+dazu/

    My colleagues from the UK team have translated it (though not the 2 updates for encrypted search and AdWords ads yet):
    http://www.dc-storm.com/news-events/471/google-announces-default-ssl-searches/

    It’s not the SSL search that removes the ‘q’ parameter value from the referrer it’s Google. It clearly shows that Google removes the parameter value for organic entries while it still pass it when clicking on and AdWords ad. That it’s not a privacy thing. Why should I remove the search term for organic clicks while still passing them for paid clicks. That doesn’t make sense.

    If the complete search process is encrypted (see https://encrypted.google.com/) the browser won’t pass any referrer at all to the target URL.

  17. Great conversation going on here :) One point of clarification and one comment:

    Clarification – this effects ALL web analytics solutions – not just GA (@Leslie, @Luca)

    Comment – both here and elsewhere people have commented on the “double standard” of doing this for organic search but not paid search, and therefore inferred an ulterior motive from Google. That is, either to pander to the needs of Adwords advertisers, or to include this as part of GA Premium (paid-for Google Analytics) i.e. “privacy is not so important if you pay us“.

    I don’t believe either to be the case. The gaping hole of showing search terms from ad click-throughs will surely be closed. Having a different approach to privacy based on what you clicked on has never been Google’s approach.

    I seriously think this is a communication mess-up – rushed and not thought through properly. As I mentioned in my post, this appears to be happening more often these days from G. For example, launching Google+ and +1 within a week of each other was confusing and misguided (was the PR team on holiday that week…?). The GA Premium product was also badly handled in my opinion.

    In fact, I find the whole idea of bombarding users with feature announcements week-on-week to be counter productive. Yes it shows products developing (good), but it also gives the impression of lack of focus and planning – as is the case here (bad).

    BTW, very good write up from Danny Sullivan here:
    http://searchengineland.com/google-to-begin-encrypting-searches-outbound-clicks-by-default-97435

  18. Luca says:

    Let’s not forget that Google Analytics is not the only free web analytics tool out there.
    My guess is that the gap left by google will be filled by other tools like Piwik. Generally speaking these tool are still a bit immature, but eventually they will catch up GA.

  19. John says:

    Google is the master of its own universe. Privacy is a great excuse to obfuscate more valuable self interests. Let’s see how loudly people roar…

  20. Tabrez Firoz says:

    Hi Brian,

    I come from a background in SEO and ask anyone from the SEO industry and they are all quite disappointed with this move. Initial reactions in twitter, suggested the same. People like Dave Naylor, Joost De Valk over at SEO Book, Danny Sullivan have all blogged about this. I suggest you have a look.

    Whats most frowned upon is that the data is available for Google’s own paid service, Google Adwords. Thats like Google saying, ‘pay me and I’ll kick privacy through the back door’.

    Some even have suggested, that may be Google would go ahead and give this data to its newly announced paid Google Analytics version, but these are just rumors and I doubt it will happen.

    As far as the impact goes, this is huge…You don’t get keyword data and hence you’ll not be able to optimize for organic search. But if you pay via Adwords, you get all the data you need. So it just implies, ‘fill my pocket & I’ll help you fill yours’. This might not be a as big as a problem to those who can afford for Adwords, but think of small businesses, your own neighborhood’s small shops who have a build website, work their socks on the data gathered from such keywords for organic traffic by working on SEO, for them things look gloomy to say the least.

    To sum it up, this is just another shot in the foot for people involved in the SEO, web analytics and say even CRO. But at the other end one can argue, this data was provided for free to begin with and now Google decides its not going to be, so just suck it up and move on.

    Then there is also the point to be considered of retargeted ads raised in SEO Book (http://www.seobook.com/false-privacy-claims). Would love to hear your thoughts on this.

    To be honest, I am yet not clearly convinced this a good move as you pointed out in your blog. If this had to be implemented, it should have also been done to Adwords data, thereby justifying the move in the right spirit. Something just does not seem right here, IMHO.

  21. Hi Brian

    Whats your thoughts of the part saying it is rolled out for google.com? I mean – how do you see the impact on searches like google.se and so?

    And yes, it is rubbish from Google. Wonder if Analytics Premium also will be able to track these searches as the only analytics tool…

  22. Rémi says:

    As Adwords can link with Google Analytics, SEO will have possibility to link with GA. In fact, Google will oblige SEO worker to adopte Google Analytics. De facto, Google is killing other webanalytics tools. Privacy is the excuse that allow these changes.

  23. James D says:

    @Leslie, it’s for all platforms – the referrer won’t be passed over at all.

    Matt Cutts has said it will be in “single digits” of Google users, but like you, I’m always signed in so I think the impact will be bigger than they think.

  24. Leslie says:

    It will be interesting to see the change in the keywords tracked versus not tracked. Is there any data out there that estimates how many users are logged in when searching? Is it 10-15% of people using Google to search or is it closer to 65-70%? That is my biggest question. Testing will tell. I know I am always logged in to my gmail, so there goes my data for anyone hoping to use it.

    Is this specifically for Google Analytics or is Google planning on blocking them for other analytics platforms as well? Hopefully the change isn’t too large otherwise clients may be frustrated.

  25. Matt Clarke says:

    I couldn’t agree more, Brian. This seems to be another hurdle for web analytics to overcome.

    Correct me if I wrong, but I thought the reason why the SSL search option was being introduced was to prevent security issues that could potentially arise when users use Google on insecure networks, such as public wifi hotspots. When the user goes from the https connection to the http sites which typically show in search results, the referrer information doesn’t get passed across. I guess, therefore, that more thought has gone into solving the potential security issue associated with that.

    Perhaps Google could have solved that by appending the keywords to the landing page URL. I think it already does something similar with Adwords traffic (gclid?) which is presumably why PPC ads aren’t affected. (If you ignore those saying this is purely about promoting the importance of PPC.)

    Judging by the massive amount of negative feedback on their blog announcement, I would hope that they might be able to do something…

    Matt

  26. James D says:

    It’s worth mentioning that Google won’t be sending referring search terms to ANY website or system when a user is logged in.

    I’ve written a blog post on our website that explains how it affects retailers and personalisation: http://www.predictiveintent.com/2011/10/what-googles-ssl-search-means-for-personalisation/.

Leave a Reply

Your email address will not be published. Required fields are marked *


9 × = 63

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

© Brian Clifton 2015
Best practice privacy statement