Accuracy Whitepaper for web analytics

This accuracy post is an extrapolation of Chapter 2 the book Advanced Web Metrics with Google Analytics, which led to a separate PDF accuracy whitepaper available to download for free. [updated April 2010 – see new post]

Download free accuracy whitepaper Why is this necessary? Well, the question of accuracy crops up all the time on numerous forums and at conferences. Essentially many practitioners of web analytics worry about accuracy. Some vendors even claim greater accuracy than others (though as I explain in the whitepaper this cannot be true), and there is the inter-industry debate about whether off-site analytics (for example, Hitwise, comScore, Neilsen//Netratings etc.), are better at predicting traffic levels than on-site analytics tools (such as Webtrends, Omniture, IndexTools, Google Analytics etc.). I won’t go into that debate here, except to schematically illustrate the two different web analytics approaches in Figure 1.

Figure 1 : On-site v off-site web analytics

Schematic showing different web analytics methodologies - taken from Adtech, London, 2007

The truth is, for either approach, web analytics is not 100% accurate, and even the error bars are difficult to measure. But just how important is this?

The initial reaction to this question is usually “very important”. Marketers need to manage visitor acquisition budgets, content creators need to be know if their work is engaging (building relationships or not), e-commerce managers need to know their conversion rates, and webmasters/developers require data on which to base decisions for technology investment (internal site search, rich media applications, checkout systems). But is this a question of accuracy or is precision the more important factor?

Wikipedia has an excellent article on the difference between accuracy vs precision and I am reliably informed that Jim Novo spoke eloquently about this at the 2007 eMetrics Summit in Washington (though I wasn’t present!). From the Wikipedia article, the two targets shown in Figure 2, schematically illustrate the difference beautifully.

Figure 2: Accuracy versus precision
accuracy v precision

This highlights that repeatable and reproducible results are what marketers, webmasters and web site owners require i.e. Precision. As long as your data reports have this, then your trends will be accurate and your decisions based on solid foundations. I emphasize the word trends above as that is the most important aspect of using web analytics – placing your data in context.

Knowing you have 10,000 visits last week is a meaningless piece of data. Knowing your visit numbers are up 10% last week to 10,000 is piece of information you can take action on. Trends (context) is the difference between yet another piece of boring data and information – which is what you use to make a decision. You should always aim for the latter.

So, back to the web analytics accuracy whitepaper . The 14 page document describes the accuracy limitations of on-site web analytics tools and how can you mitigate these and get comfortable with your data. Importantly, it is vendor agnostic. That is, with a best practice implementation of your web analytics tool, you can get very precise visitor data.

I hope you find it useful as a reference guide for practitioners and clients alike. If you would like to share your thoughts on the document, please add a comment.

Looking for a keynote speaker, or wish to hire Brian…?

If you are an organisation wishing to hire me and my team, please view the Contact page. I am based in Sweden and advise organisations in Europe as well as North America.

You May Also Like…

11 Comments

  1. Bruno

    Brian, Thanks for the whitepaper.

    Am curious whether there are any more up to date (and in-depth) research projects that analyze the differences between analytics softwares.

    Reply
    • Brian Clifton

      Bruno – I am not aware of anything written more recently on this subject, I suspect because accuracy issues have not really changed. GA has certainly moved on so I will review my doc and see what updates can be applied. Otherwise I would be interested in your feedback on anything you feel is missing or out of date.

      Reply
  2. Rob W

    Great Whitepaper! I feel like printing out 20 copies and just giving them to clients that get “hung up” on asking why their website analytics data isn’t 100% accurate.

    Reply
    • Brian Clifton

      RobW: That is such great feedback Rob – you are obviously reading the latest version which I silently launched last month. I will be submitting a new post article for the latest version next week and we put your comments there also. Once again, many thanks! That’s exactly why I write such articles 🙂

      [edit]
      Here’s the updated link for latest accuracy whitepaper post.

      Reply
  3. Martijn Staal

    Brian, Thanks for the whitepaper. Interesting view on the accuracy of web analytics.

    Some time ago we also discovered that there are a lot of differences between the metrics that are used by web analytics software. For example, Google Analytics measures a unique visitor in a different way then DART or Sitestat. In January I also posted an article about the differences between analytics software (unfortunately in Dutch): http://www.marketingfacts.nl/berichten/20080115_hoe_betrouwbaar_zijn_website_statistieken/

    Together with multiple marketing agencies we are now working on a pilot to compare the desired analytics functionality against the functionality that is being provided by current analytics software.

    We know about some other research at http://www.seomoz.org/blog/how-reliable-is-google-analytics, http://www.stonetemple.com/articles/analytics-report-may-2007.shtml and http://www.idealware.org/blog/2007/06/google-analytics-underreporting-or-just.html.

    Am curious whether you know any more research projects in which the differences between analytics software have been investigated. Do you, or any of the other visitors here, know about similar research?

    Reply
  4. Ryan

    Great whitepaper! Thanks.

    Reply
  5. Brian Clifton

    Steve/Chris – thanks for the feedback. I hope its of use at providing answers to those never ending questions about accuracy and data alignment. Its helped me a great deal to collate this into one document – I wish I had written it years ago. In fact its been bouncing around for a least two years in one form or another…

    Jim: No marketer works with accurate numbers, so it should not be a great surprise that web analytics is not accurate. In fact, web analytics is several orders of magnitude more accurate than the offline world, its just not 100% accurate, which is a common misconception for the reasons I give.

    Sara: You ask a good (and difficult) question. Let me ponder this for a while with some fictitious examples and post a reply.

    Reply
  6. Sara

    I am happy to see that an analyst is talking about trends and although I agree with the fact that accuracy is important and of course ideal – it is the truth and a fact that we have to live with.

    Looking at it from a marketer perspective I know that there are way to many sources of data or studies that don’t actually even predict what the result will be. Many times we conduct focus group studies just to see… what likelihoods we have of hitting the right target group with the right product or the right message. We know accuracy only comes down to sales and this is where we need to look for trends. Bottom line is what is important and to make decisions in a marketing world full of not only web analytics but tons of data sources we need to follow the trends.

    Most sites are not conducting e-commerce and we need to spend our resources analysing the lead generations through a combination of the online traffic to the other trend tools available.

    A question for you: – In your opinion..how would you best go about to study a trend when you have to aggregate additional resources of information from various tools (all with their accuracy problems)?

    Reply
  7. Jim

    Hi Brian – Thanks for the whitepaper, looks like a good summary of common accuracy issues. My concern is that although the ideal of precision being more important than accuracy is one that all analysts (in particular when faced with conflicting data) should espouse, this doesn’t always work in the real world. Out in the trenches, stakeholders who rely on analytics data to make business decisions often need to make these decisions based on numbers that need to be accurate, for example when pricing CPM ads or developing business models around traffic numbers.

    Explaining to them that the numbers may not be accurate but they are precise is going to get you some very odd looks, to say the least…

    Reply
  8. Chris Grant

    Nice white paper – systematic, thorough, and articulate. Thanks!

    Reply
  9. Steve Jackson

    Very useful whitepaper thanks Brian. It helps clarify the issues that practitioners face.

    Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Share This