Why counting uniques is meaningless

The term ‘uniques’ is often used in web analytics as an abbreviation for unique web visitors (i.e. how many unique people visited my site). The problem is that counting unique visitors is fraught with problems that are so fundamental, it renders the term ‘uniques’ meaningless.

Firstly, cookies get lost, blocked and deleted. Research has shown that after a period of four weeks, nearly one third of tracking cookies are missing, which means the visitor will be incorrectly considered a new unique visitor should they return to the same website (see Accuracy Whitepaper for further reading).

The longer the time period, the greater the chance of this happening, which makes comparing year-on-year data invalid for example. In addition, browsers make it very easy these days for cookies to be removed – see the new ‘incognito’ features of the latest Firefox, Chrome and Internet Explorer browsers.

However, the biggest issue for counting uniques faced by both on and off-site web analytics tools is how many devices people use to access the web. For example, consider the following scenario:

You and your spouse are considering your next holiday. Your spouse first checks out possible locations on your joint PC at home and saves a list of website links.

The next evening you use the same PC to review these links. Unable to decide that night, you email the list to your office and the next day you continue your holiday checks during your lunch hour at work and also review these again on your mobile while commuting home on the train.

Day three of your search resumes at your friend’s house where you seek a second opinion. Finally you go home and book online using your shared PC.

The above scenario is actually very common – particularly if the value of the purchase is significant, which implies a longer consideration period and the seeking of a second opinion (spouse, friends work colleagues).

Simply put, there is not a web analytics solution in the world that can accurately track this scenario, that is to tie the data together from multiple devices and where multiple people have been involved, nor is there likely to be in the near future.

Combining these limitations leads to large error bars when it comes to tracking uniques. In fact these errors are so large that the metric is actually meaningless and should be avoided in favour of more accurate ‘visit’ data.

Update (Apr-09): coincidentally Eric Peterson also posted a much longer blog article on the same issues in March – here’s the link: http://blog.webanalyticsdemystified.com/weblog/2009/03/unique-visitors-only-come-in-one-size.html

Looking for a keynote speaker, or wish to hire Brian…?

If you are an organisation wishing to hire me and my team, please view the Contact page. I am based in Sweden and advise organisations in Europe as well as North America.

You May Also Like…

25 Comments

  1. Pete

    I would like to see this revisited after Google’s combining all a users accounts and data into one – in my case it has taken my youtube, mobile, personal and work email data and linked GA accounts and merged it into one ‘super-account’.

    As a member of the public I find this ghastly, but as an eCommerce expert I would be interested to see if google has overcome this problem in the given scenario -whereby a user on multiple devices can be tracked – in a more meaningful way?

    Reply
  2. Mark

    Brian,

    yeh, sorry I was trying to be brief. And as you know this topic generates a lot of heat.

    As for seven visits being seven opportunities to sell, I sort of agree, but i would make the point it depends on what you’re selling.

    If it’s a low value item then probably it is seven opportunities to sell seven different items.

    If it’s a high value item, say a house, then it’s seven opportunities to build a relationship and probably only one opportunity for a sale.

    And the business owner will want to know if they are dealing with one person or seven.

    And as I said, no matter how you dress it up and explain it, management and business owners will take whatever number you give them as “real people” They always have.

    I fully understand and accept your point about the data accuracy issue, but it’s view taken from a technical standpoint, I might even say left brain, methodical type.

    The marketer and business owner wants to know the number of real people, which is more right brain, intuitive, humanistic. (Ooops, I’ve probably started a whole ‘nother debate now.)

    As I keep implying we are never going to “educate them and move on” about the distinction between uniques and visits, because they are are trying to educate us about what they want, not what we can give them.

    In any event, let’s not get too obsessed over conversion rate, because it is in itself a very flawed metric.

    A site with a 2% conversion rate can easily be a much better performing site than one with a 20% conversion rate.

    We do need to move on to a metric or set of metric that gives a more accurate picture of how a website is performing.

    Reply
  3. Mark

    Oh, and one more thing. I think if everyone started using Visits as the basis for their conversion rate calcualtion, the global average conversion rate would be well below 1%.

    Reply
  4. Mark

    I don’t think this argument is going to be resolved anytime soon.

    While I understand the logic of why Visits is a more accurate measure, I disagree with favouring it over Uniques.

    Visits is accurate from a technical standpoint, but as marketers and business owners we want to know how many customers (people) visited our sites and bought from us. And in that regard Visits is obviously a useless metric.

    We don’t want one customer who made seven visits counted as seven people.

    Uniques is closer to the mark than Visits even with all it’s flaws listed above, but at present it’s the best we’ve got.

    If we were to take this argument to it’s logical conclusion then all analytics packages are useless because none of them can tell us how many real people actually came to our website.

    But I digress.

    The other thing to remember is that it doesn’t matter what term you tell management they will interpret it as real people numbers.

    Surely most of you will remember when “hits” was used to mean real people, until most of us found out it was counting all the elements on a page etc, etc.

    The point is, you’re not going to change how management hears it, no matter how detailed your explanations, because it’s not what they want to hear.

    As I’ve said, they want to know real people numbers.

    But fear not, there is a much more definitive metric you can use that everyone will understand, management will love and there will be no confusion about:

    “Our site did $X00,000 last month.”

    Now that’s a metric, and one that the bank manager will gladly accept.

    Reply
    • Brian Clifton

      Mark: thanks for your input on this discussion. I would be interested to know why the following statement from you is so important – “We don’t want one customer who made seven visits counted as seven people.

      That is 7 opportunities to sell/build a relationship – what difference does it make if it is the same person, or seven different people? Of course there is a difference to the business. You want to know if you have a repeat, loyal customers – and your e-commerce reports will tell you this. However, form a website conversion optimisation point of view, it is not important to make that differentiation.

      In an idea world, yes we would use unique visitors – but that isn’t the case, and more importantly it *never* will be. So should we stick with a flawed metric or educate users to move on? The point of this post was to add a vote (and the reasoning behind it) to the latter. That is, move away from unique visitors*.

      *If you are a website that requires a log-in to access/purchase content, then it still makes sense to use unique visitors. For example, publishing or an ebay, amazon type of site.

      BTW, I certainly prefer revenue related metrics as KPIs to any others. Even if not a transactional site, “value” should be caluclated where possible. A couple of relevant posts on this:

      Reply
  5. AnalyticsGirl

    @brianclifton Thanks for this – I’m sending it straight to a client. Saves me explaining it.

    Reply
  6. Ralph Jones

    To be honest I really never gave thought at looking at unique visitor counts from this aspect and I agree with you on this as I myself have been a part of the example that you mentioned and from my experience, most cookies get lost in a week and much faster if a person has a habit of deleting them by pc tools everyday.

    Reply
  7. Ace

    Thanks a lot…it have widened my horizons….im a beginner with analytics and i found good knowledge on this subject..Thank You

    Reply
  8. BClifton

    Robert: this may not be so far fetched. I am thinking of the Open Social project initiated by Google a couple of years ago…

    Essentially the idea is that rather than users having to create and manage multiple social networking “profiles” with a plethora of sites, you have one profile and simply enable/disable sharing of this information with the new site you join.

    I can imagine this expanding to all sites and hence the use of “uniques” becomes much more valid. But it all hinges on trust. That is, the user has to trust you (the website owner) to handle their information with care/respect. See my related post entitled Google is Like a Bank (probably not the analogy they want to hear in the current climate!).

    Care/respect of privacy appears to be a rare commodity on the web. I put Google and Firefox at the top of my list for this. But my list ends at those two. I don’t think I trust anyone else to uphold the un-written standard. One thing is for sure, those sites that respect end-user privacy will be the winners in the end…

    Reply
  9. Robert K

    Totally agree – unique visitors is a strange thing – cookies/IP addresses being tracked doesnt make the user a unique visitor – ip addresses are recycled and as mentioned in the post cookies are rarely permenantly stored.

    As far as I can see there is no way around this unless in the future (nanny-states etc – or even 1984 style) everyone has a unique trackable address assigned to them from birth – sound ridiculous? Probably, but who knows how the interweb will be in 5 years time?

    Reply
  10. BClifton

    Jesse: I am going to use some borrowed wisdom from an author I am currently reading. To paraphrase him:

    Logic does not require empirical verification. It is a mistake to use statistics without logic, but the reverse does not hold true: It is not a mistake to use logic without statistics.

    Gerry: see my comment on this “common yard stick” approach in comment #8

    Reply
  11. Gerry White

    Does it matter that it isn’t quite right ?

    My argument on this is that conversion on a per visit basis is sometimes not as useful as a visitor basis – we can’t be accurate but it can be a basis for a KPI – i.e. a number we can use to improve upon. If it is for an unconsidered quick purchase – yes, visits, otherwise visitors (example is banking products, particularly ones where you don’t usually buy more than one of them).

    I am presently car hunting, so me going back to the same website, but not buying, is actually a positive thing (if I do buy from that site in the end).

    Reply
  12. Jesse Farmer

    Brian,

    I guess my challenge to you is this: do you have any data to back up your argument? You say “the error bars are too large,” but that’s it. You can give as many reasons you want why this *might* be so, but without actual data — ideally data that caused someone to make an incorrect decision — it’s just your word against a whole family of plausible alternatives.

    Reply
  13. Ole Bahlmann

    Hi Brian,
    I enjoyed this post of yours very much although I’m more inclined to agree with Jesses last Comment.

    But there is one thing that struck me instantly after reading:
    “Research has shown that after a period of four weeks, nearly one third of tracking cookies are missing, […]”

    I’m very interested into these kinds of research and be grateful if you’d share your source.

    Thanks,
    Ole

    Reply
  14. Brian Clifton

    Jesse: you make a valid point regarding using the same “yard stick” i.e. as long as the measuring device remains the same, then the difference or trends of metrics will be accurate. So a 10% increase in visitor traffic will be accurate even if the absolute number is inaccurate.

    However, although that is valid for the vast majority of metrics, the exception is Unique Visitors. As illustrated in the reasons I give above, the error bars associated with unique visitors are random – there really is no way to determine the average ‘cookie loss’ rate for a site, or how often visitors use an alternative device to connect. Unless…

    Unless there are strong reasons for visitors to identify themselves such as via a login or loyalty card scheme (see comment #2).

    This will work for a particular business sector in a specific market. However, those numbers are not valid beyond that.

    Tim’s response has given me an idea (comment #6) that I hope to test next quarter…

    Reply
  15. Jesse Farmer

    I agree that the way most web analytics packages measure “uniques” has some level of error, but I wouldn’t go so far to call the number “meaningless.”

    So long as we account for the error (uncertainty) in our decision I see no problem. Statistics gives us a a whole rigorous toolset for doing just this.

    Tools like Google Analytics could do a better job of exposing the uncertainty, perhaps showing upper and lower bounds for the metrics you graph, but it’s still better than nothing.

    You say, “Combining these limitations leads to large error bars when it comes to tracking uniques. In fact these errors are so large that the metric is actually meaningless and should be avoided in favour of more accurate ‘visit’ data.”

    Do you have anything to back this up? Maybe a spreadsheet that lays out some scenarios where the “error bars” are too large? More accurate measurements are always better, but you don’t always need to be operating at micrometer-level precision, if you know what I mean. Sometimes it’s ok to be measuring by the meter.

    Reply
  16. Brian Clifton

    Tim: That’s an excellent idea and has got me thinking…

    I have a client that has a long, convoluted registration process that must be completed off-line for the user to gain access (several days, if not weeks). All of those visitors that do sign in therefore must be returning visitors.

    Looking at how many of these actually show as new visitors will give us a handle of the size of the problem I describe.

    Unfortunately the client login area is managed by a different department and development agency, so tracking this is not currently in place. However, it should be straight forward for me to get this done.

    Looking forward to posting back on this one…

    Reply
  17. Tim Leighton-Boyce

    @Bjoern: I was thinking of a specific group of e-commerce sites on which I work. When a returning user logs in they see a different page from someone who is shopping for the first time. We use this in many ways — separate funnel reports, for example.

    My general point is that if you use any reporting method which allows you to pinpoint a group of visits which you know to be ‘returning’ and then look at the new vs returning split, you will get an insight into the reliability of ‘new/returning’, ‘uniques’ and so on.

    The discrepancy will vary according to the behaviour of the users of the site (their likelihood to use multiple systems for accessing the site) of course. I doubt if there is any usable average. Which is why I am very wary of reports which give naive users the impression that they are looking at unique visitors. Perhaps the reports should be called something which signals the need for careful interpretation. Or carry a health warning.

    Reply
  18. Bjoern

    >According to the standard reports, many of the people who
    >log in will be ‘new’. Which they are clearly not.

    That’s a good workaround, I think. But wouldn’t people that sign up and log into the site in one visit be correctly identified as “new”?

    Maybe it would be possible to setup a userdefined variable for “registered users” that kicks in, when a user logs in and his account creation date is older than one day.

    That way it might be possible to get an even more exact idea of the “Pinch of salt”..

    Reply
  19. Tim Leighton-Boyce

    If you have a site which has a ‘log in’ option, then one way of getting a feel for the scale of the problem on your site is to have a look at the split between ‘new’ and ‘returning’ visitors for people who log in.

    According to the standard reports, many of the people who log in will be ‘new’. Which they are clearly not.

    You then have a rough idea of the size of pinch of salt to be taken with such reports.

    Reply
  20. Brian Clifton

    Bjoern: There are scenarios where web analytics tools can accurately measure “uniques” though these are rare. For example, online banking requires visitors to login and these details are almost never shared – so you have a very accurate measure of the number of people using your site.

    Other examples include popular brands such as Amazon, FedEx, social network sites (mySpace, Facebook et al), where there is a strong reason to a) have an account with the web site in the first place and b) significant benefit to being logged in even if you do not wish to purchase/transact.

    Another strong brand where this works is for Tesco (largest supermarket chain in the UK by revenue). They have a loyalty card, called Club Card, that builds points as you shop and can be exchanged for goods. Club card members get special offers and discounts not available to non-members.

    So there is always a strong desire on the part of the Tesco web visitor to use their unique Club Card number during their session and hence generate accurate unique visitor traffic.

    However, these exceptions are very rare…

    Reply
  21. Bjoern

    I totally agree that the data web analytics software can provide as “uniques” is useless and misleading.

    However, there are approaches to measure unique users in more complex scenarios, primarily to identify the real number of users (as in “people”) and their sociodemography for advertising related purposes.

    In Germany this kind of audience reasearch is conducted by the AGOF and their “Internet Facts Study”: It’s a 3-method approach, including on-site-tracking, on-site-surveys and massive, statistically representative offline CATI interviews. Here is an outline of the methodology in German: http://www.agof.de/methode.585.html

    Reply

Leave a Reply to BClifton Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Share This