This accuracy post is an extrapolation of Chapter 2 the book Advanced Web Metrics with Google Analytics, which led to a separate PDF accuracy whitepaper available to download for free. [updated April 2010 – see new post]
Why is this necessary? Well, the question of accuracy crops up all the time on numerous forums and at conferences. Essentially many practitioners of web analytics worry about accuracy. Some vendors even claim greater accuracy than others (though as I explain in the whitepaper this cannot be true), and there is the inter-industry debate about whether off-site analytics (for example, Hitwise, comScore, Neilsen//Netratings etc.), are better at predicting traffic levels than on-site analytics tools (such as Webtrends, Omniture, IndexTools, Google Analytics etc.). I won’t go into that debate here, except to schematically illustrate the two different web analytics approaches in Figure 1.
Figure 1 : On-site v off-site web analytics
The truth is, for either approach, web analytics is not 100% accurate, and even the error bars are difficult to measure. But just how important is this?
The initial reaction to this question is usually “very important”. Marketers need to manage visitor acquisition budgets, content creators need to be know if their work is engaging (building relationships or not), e-commerce managers need to know their conversion rates, and webmasters/developers require data on which to base decisions for technology investment (internal site search, rich media applications, checkout systems). But is this a question of accuracy or is precision the more important factor?
Wikipedia has an excellent article on the difference between accuracy vs precision and I am reliably informed that Jim Novo spoke eloquently about this at the 2007 eMetrics Summit in Washington (though I wasn’t present!). From the Wikipedia article, the two targets shown in Figure 2, schematically illustrate the difference beautifully.
Figure 2: Accuracy versus precision
This highlights that repeatable and reproducible results are what marketers, webmasters and web site owners require i.e. Precision. As long as your data reports have this, then your trends will be accurate and your decisions based on solid foundations. I emphasize the word trends above as that is the most important aspect of using web analytics – placing your data in context.
Knowing you have 10,000 visits last week is a meaningless piece of data. Knowing your visit numbers are up 10% last week to 10,000 is piece of information you can take action on. Trends (context) is the difference between yet another piece of boring data and information – which is what you use to make a decision. You should always aim for the latter.
So, back to the web analytics accuracy whitepaper . The 14 page document describes the accuracy limitations of on-site web analytics tools and how can you mitigate these and get comfortable with your data. Importantly, it is vendor agnostic. That is, with a best practice implementation of your web analytics tool, you can get very precise visitor data.
I hope you find it useful as a reference guide for practitioners and clients alike. If you would like to share your thoughts on the document, please add a comment.