Accuracy Whitepaper for web analytics
This accuracy post is an extrapolation of Chapter 2 the book Advanced Web Metrics with Google Analytics, which led to a separate PDF accuracy whitepaper available to download for free. [updated April 2010 – see new post]
Why is this necessary? Well, the question of accuracy crops up all the time on numerous forums and at conferences. Essentially many practitioners of web analytics worry about accuracy. Some vendors even claim greater accuracy than others (though as I explain in the whitepaper this cannot be true), and there is the inter-industry debate about whether off-site analytics (for example, Hitwise, comScore, Neilsen//Netratings etc.), are better at predicting traffic levels than on-site analytics tools (such as Webtrends, Omniture, IndexTools, Google Analytics etc.). I won’t go into that debate here, except to schematically illustrate the two different web analytics approaches in Figure 1.
Figure 1 : On-site v off-site web analytics
The truth is, for either approach, web analytics is not 100% accurate, and even the error bars are difficult to measure. But just how important is this?
The initial reaction to this question is usually “very important”. Marketers need to manage visitor acquisition budgets, content creators need to be know if their work is engaging (building relationships or not), e-commerce managers need to know their conversion rates, and webmasters/developers require data on which to base decisions for technology investment (internal site search, rich media applications, checkout systems). But is this a question of accuracy or is precision the more important factor?
Wikipedia has an excellent article on the difference between accuracy vs precision and I am reliably informed that Jim Novo spoke eloquently about this at the 2007 eMetrics Summit in Washington (though I wasn’t present!). From the Wikipedia article, the two targets shown in Figure 2, schematically illustrate the difference beautifully.