Noise or Music?

Accuracy Whitepaper for web analytics

Categories: GDPR & Privacy, Metrics Understanding, Setup Accuracy / Comments: 12

This accuracy post is an extrapolation of Chapter 2 the book Advanced Web Metrics with Google Analytics, which led to a separate PDF accuracy whitepaper available to download for free. [updated April 2010 – see new post]

Download free accuracy whitepaper Why is this necessary? Well, the question of accuracy crops up all the time on numerous forums and at conferences. Essentially many practitioners of web analytics worry about accuracy. Some vendors even claim greater accuracy than others (though as I explain in the whitepaper this cannot be true), and there is the inter-industry debate about whether off-site analytics (for example, Hitwise, comScore, Neilsen//Netratings etc.), are better at predicting traffic levels than on-site analytics tools (such as Webtrends, Omniture, IndexTools, Google Analytics etc.). I won’t go into that debate here, except to schematically illustrate the two different web analytics approaches in Figure 1.

Figure 1 : On-site v off-site web analytics

Schematic showing different web analytics methodologies - taken from Adtech, London, 2007

The truth is, for either approach, web analytics is not 100% accurate, and even the error bars are difficult to measure. But just how important is this?

The initial reaction to this question is usually “very important”. Marketers need to manage visitor acquisition budgets, content creators need to be know if their work is engaging (building relationships or not), e-commerce managers need to know their conversion rates, and webmasters/developers require data on which to base decisions for technology investment (internal site search, rich media applications, checkout systems). But is this a question of accuracy or is precision the more important factor?

Wikipedia has an excellent article on the difference between accuracy vs precision and I am reliably informed that Jim Novo spoke eloquently about this at the 2007 eMetrics Summit in Washington (though I wasn’t present!). From the Wikipedia article, the two targets shown in Figure 2, schematically illustrate the difference beautifully.

…continue reading ‘Accuracy Whitepaper for web analytics’

Share Button

Comments (most recent first)

  1. Bruno says:

    Brian, Thanks for the whitepaper.

    Am curious whether there are any more up to date (and in-depth) research projects that analyze the differences between analytics softwares.

    • Bruno – I am not aware of anything written more recently on this subject, I suspect because accuracy issues have not really changed. GA has certainly moved on so I will review my doc and see what updates can be applied. Otherwise I would be interested in your feedback on anything you feel is missing or out of date.

  2. Rob W says:

    Great Whitepaper! I feel like printing out 20 copies and just giving them to clients that get “hung up” on asking why their website analytics data isn’t 100% accurate.

    • RobW: That is such great feedback Rob – you are obviously reading the latest version which I silently launched last month. I will be submitting a new post article for the latest version next week and we put your comments there also. Once again, many thanks! That’s exactly why I write such articles 🙂

      Here’s the updated link for latest accuracy whitepaper post.

  3. Brian, Thanks for the whitepaper. Interesting view on the accuracy of web analytics.

    Some time ago we also discovered that there are a lot of differences between the metrics that are used by web analytics software. For example, Google Analytics measures a unique visitor in a different way then DART or Sitestat. In January I also posted an article about the differences between analytics software (unfortunately in Dutch):

    Together with multiple marketing agencies we are now working on a pilot to compare the desired analytics functionality against the functionality that is being provided by current analytics software.

    We know about some other research at, and

    Am curious whether you know any more research projects in which the differences between analytics software have been investigated. Do you, or any of the other visitors here, know about similar research?

  4. Ryan says:

    Great whitepaper! Thanks.

  5. Steve/Chris – thanks for the feedback. I hope its of use at providing answers to those never ending questions about accuracy and data alignment. Its helped me a great deal to collate this into one document – I wish I had written it years ago. In fact its been bouncing around for a least two years in one form or another…

    Jim: No marketer works with accurate numbers, so it should not be a great surprise that web analytics is not accurate. In fact, web analytics is several orders of magnitude more accurate than the offline world, its just not 100% accurate, which is a common misconception for the reasons I give.

    Sara: You ask a good (and difficult) question. Let me ponder this for a while with some fictitious examples and post a reply.

  6. Sara says:

    I am happy to see that an analyst is talking about trends and although I agree with the fact that accuracy is important and of course ideal – it is the truth and a fact that we have to live with.

    Looking at it from a marketer perspective I know that there are way to many sources of data or studies that don’t actually even predict what the result will be. Many times we conduct focus group studies just to see… what likelihoods we have of hitting the right target group with the right product or the right message. We know accuracy only comes down to sales and this is where we need to look for trends. Bottom line is what is important and to make decisions in a marketing world full of not only web analytics but tons of data sources we need to follow the trends.

    Most sites are not conducting e-commerce and we need to spend our resources analysing the lead generations through a combination of the online traffic to the other trend tools available.

    A question for you: – In your would you best go about to study a trend when you have to aggregate additional resources of information from various tools (all with their accuracy problems)?

  7. Jim says:

    Hi Brian – Thanks for the whitepaper, looks like a good summary of common accuracy issues. My concern is that although the ideal of precision being more important than accuracy is one that all analysts (in particular when faced with conflicting data) should espouse, this doesn’t always work in the real world. Out in the trenches, stakeholders who rely on analytics data to make business decisions often need to make these decisions based on numbers that need to be accurate, for example when pricing CPM ads or developing business models around traffic numbers.

    Explaining to them that the numbers may not be accurate but they are precise is going to get you some very odd looks, to say the least…

  8. Chris Grant says:

    Nice white paper – systematic, thorough, and articulate. Thanks!

  9. Very useful whitepaper thanks Brian. It helps clarify the issues that practitioners face.

  1. […] not what you want to hear if you are the end-user trying to interpret your visitor reports! The original accuracy whitepaper (published in Feb 2008) explained all of the possible accuracy considerations I could think of at […]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© Brian Clifton 2021
Best practice privacy statement