Noise or Music? - The Insights Blog

Accuracy Whitepaper for web analytics

February 16, 2008 / Categories: Metrics understanding, Privacy and Accuracy / Comments: 10

Share Button

This accuracy post is an extrapolation of a section in Chapter 2 of the book which has led to a separate PDF accuracy whitepaper available to download for free. [updated April 2010 – see new post]

Download free accuracy whitepaper Why is this necessary? Well, the question of accuracy crops up all the time on numerous forums and at conferences. Essentially many practitioners of web analytics worry about accuracy. Some vendors even claim greater accuracy than others (though as I explain in the whitepaper this cannot be true), and there is the inter-industry debate about whether off-site analytics (for example, Hitwise, comScore, Neilsen//Netratings etc.), are better at predicting traffic levels than on-site analytics tools (such as Webtrends, Omniture, IndexTools, Google Analytics etc.). I won’t go into that debate here, except to schematically illustrate the two different web analytics approaches in Figure 1.

Figure 1 : On-site v off-site web analytics

Schematic showing different web analytics methodologies - taken from Adtech, London, 2007

The truth is, for either approach, web analytics is not 100% accurate, and even the error bars are difficult to measure. But just how important is this?

The initial reaction to this question is usually “very important”. Marketers need to manage visitor acquisition budgets, content creators need to be know if their work is engaging (building relationships or not), e-commerce managers need to know their conversion rates, and webmasters/developers require data on which to base decisions for technology investment (internal site search, rich media applications, checkout systems). But is this a question of accuracy or is precision the more important factor?

Wikipedia has an excellent article on the difference between accuracy vs precision and I am reliably informed that Jim Novo spoke eloquently about this at the 2007 eMetrics Summit in Washington (though I wasn’t present!). From the Wikipedia article, the two targets shown in Figure 2, schematically illustrate the difference beautifully.

Figure 2: Accuracy versus precision
accuracy v precision

This highlights that repeatable and reproducible results are what marketers, webmasters and web site owners require i.e. Precision. As long as your data reports have this, then your trends will be accurate and your decisions based on solid foundations. I emphasize the word trends above as that is the most important aspect of using web analytics – placing your data in context.

Knowing you have 10,000 visits last week is a meaningless piece of data. Knowing your visit numbers are up 10% last week to 10,000 is piece of information you can take action on. Trends (context) is the difference between yet another piece of boring data and information – which is what you use to make a decision. You should always aim for the latter.

So, back to the web analytics accuracy whitepaper . The 14 page document describes the accuracy limitations of on-site web analytics tools and how can you mitigate these and get comfortable with your data. Importantly, it is vendor agnostic. That is, with a best practice implementation of your web analytics tool, you can get very precise visitor data.

I hope you find it useful as a reference guide for practitioners and clients alike. If you would like to share your thoughts on the document, please add a comment.

Share Button

Comments

  1. watchstore says:

    Good post, the pictures seems beautiful. my dream is go travel around the world. I belive my dream will come true one day. From this, I know other countries costums. I like this kinds life. Jaeger Watch

  2. Rob W says:

    Great Whitepaper! I feel like printing out 20 copies and just giving them to clients that get “hung up” on asking why their website analytics data isn’t 100% accurate.

    • RobW: That is such great feedback Rob – you are obviously reading the latest version which I silently launched last month. I will be submitting a new post article for the latest version next week and we put your comments there also. Once again, many thanks! That’s exactly why I write such articles :)

      [edit]
      Here’s the updated link for latest accuracy whitepaper post.

  3. Brian, Thanks for the whitepaper. Interesting view on the accuracy of web analytics.

    Some time ago we also discovered that there are a lot of differences between the metrics that are used by web analytics software. For example, Google Analytics measures a unique visitor in a different way then DART or Sitestat. In January I also posted an article about the differences between analytics software (unfortunately in Dutch): http://www.marketingfacts.nl/berichten/20080115_hoe_betrouwbaar_zijn_website_statistieken/

    Together with multiple marketing agencies we are now working on a pilot to compare the desired analytics functionality against the functionality that is being provided by current analytics software.

    We know about some other research at http://www.seomoz.org/blog/how-reliable-is-google-analytics, http://www.stonetemple.com/articles/analytics-report-may-2007.shtml and http://www.idealware.org/blog/2007/06/google-analytics-underreporting-or-just.html.

    Am curious whether you know any more research projects in which the differences between analytics software have been investigated. Do you, or any of the other visitors here, know about similar research?

  4. Ryan says:

    Great whitepaper! Thanks.

  5. Steve/Chris – thanks for the feedback. I hope its of use at providing answers to those never ending questions about accuracy and data alignment. Its helped me a great deal to collate this into one document – I wish I had written it years ago. In fact its been bouncing around for a least two years in one form or another…

    Jim: No marketer works with accurate numbers, so it should not be a great surprise that web analytics is not accurate. In fact, web analytics is several orders of magnitude more accurate than the offline world, its just not 100% accurate, which is a common misconception for the reasons I give.

    Sara: You ask a good (and difficult) question. Let me ponder this for a while with some fictitious examples and post a reply.

  6. Sara says:

    I am happy to see that an analyst is talking about trends and although I agree with the fact that accuracy is important and of course ideal – it is the truth and a fact that we have to live with.

    Looking at it from a marketer perspective I know that there are way to many sources of data or studies that don’t actually even predict what the result will be. Many times we conduct focus group studies just to see… what likelihoods we have of hitting the right target group with the right product or the right message. We know accuracy only comes down to sales and this is where we need to look for trends. Bottom line is what is important and to make decisions in a marketing world full of not only web analytics but tons of data sources we need to follow the trends.

    Most sites are not conducting e-commerce and we need to spend our resources analysing the lead generations through a combination of the online traffic to the other trend tools available.

    A question for you: – In your opinion..how would you best go about to study a trend when you have to aggregate additional resources of information from various tools (all with their accuracy problems)?

  7. Jim says:

    Hi Brian – Thanks for the whitepaper, looks like a good summary of common accuracy issues. My concern is that although the ideal of precision being more important than accuracy is one that all analysts (in particular when faced with conflicting data) should espouse, this doesn’t always work in the real world. Out in the trenches, stakeholders who rely on analytics data to make business decisions often need to make these decisions based on numbers that need to be accurate, for example when pricing CPM ads or developing business models around traffic numbers.

    Explaining to them that the numbers may not be accurate but they are precise is going to get you some very odd looks, to say the least…

  8. Chris Grant says:

    Nice white paper – systematic, thorough, and articulate. Thanks!

  9. Very useful whitepaper thanks Brian. It helps clarify the issues that practitioners face.

Leave a Reply

Your email address will not be published. Required fields are marked *


6 + = 14

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

© Brian Clifton 2015
Best practice privacy statement