Noise or Music? - The Insights Blog

When Voice of Customer Surveys can Damage Your Brand

Categories: Metrics understanding / Comments: 35

Share Button

Voice of customer This year’s buzz word in the world of web analytics is “Voice of Customer” or VOC for short. Essentially this boils down to presenting a survey/questionnaire to your web visitors asking them to respond to questions that can be used to ascertain how they feel about the web experience they have had.

Why voice of customer surveys are so useful

As you are no doubt aware, web analytics tools and methodologies are great for telling you the “what” and the “when” of your web site visitors. That is, what happened (a goal conversion event, a transaction, a specific pageview or a combination of pageviews etc. or any kind of engagement on your site) and when it happened (time/date, do they repeat the same thing over again and at what frequency etc.). This is quantitative data that is invaluable for identifying poor performing pages and poorly targeted marketing campaigns.

However, the missing link from web analytics has always been the qualitative data – the “why”. For example, why did 30% of visitors leave your web site on that particular page, only view one page (bounce), not convert, not buy, not contact you, stay for less than 10 seconds etc. The only way to obtain such information is to ask them why.

In fact, without asking your visitors, ascertaining qualitative information is often guess work. For example, following a site redesign, why has your time on site metric increased? Is it because users are more engaged with your content, or is it because they are lost in your new navigation layout? Taking a guess can lead to a very invalid assumption about your visitor satisfaction.

Allowing visitors to quickly and easily provide their feedback bridges the gap between anonymous, aggregate web analytics data (traffic) and the views of your visitors (personalised responses from individuals). As an aside, I strongly recommend that personalised responses also remain anonymous for best practice privacy reasons, unless of course the visitor expressly wishes to give their personal information.

When voice of customer surveys can damage your brand

So with so much to benefit from deploying a VOC survey, what can be bad…?

What I find quite astounding is the current trend to use pop-up windows as the method of survey deployment. That is, an unsolicited window pops up in front of the visitor (usually selecting visitors at random) requesting them to participate in a survey. By unsolicited, I mean the opening of an additional window without any action or knowledge by the visitor. This is not the same as a visitor’s action opening a new window – for example, the clicking a link or button.

Pop-up windows are one of the oldest and most annoying forms of interruption marketing on the web – a phrase I borrow from Seth Godin (whose books I highly recommend for viewing the bigger picture of digital marketing).

They are so annoying that they sparked the creation of a whole industry of anti-popup and ad blocking software in the late 90s and early 2000s. In fact, the pop-up blocking capabilities of the Google Toolbar was a main reason for its success – one of the most popular software releases ever, with an estimated 100+ million downloads to date.

Unsolicited pop-ups are similar to email spam. That is, hated by Internet users. Their use to display a survey/questionairre, not only puts you in the category of a spammer – annoying your visitors, it is also likely to skew results. For example, consider the following scenario:

A happy visitor on your site enjoying their user experience becomes irritated with an unsolicited pop-up requesting their feedback. They either leave your web site, decreasing your survey participation rate, or provide negative feedback because of the annoyance.

Of course, it is possible they welcome the option to provide feedback, but how likely is that in reality? Probably just as likely that they would want to receive a spam email from you. It happens, but it’s so rare that that damage to your brand caused by the interruption, far out weighs any feedback gain.

For this reason, survey participation rates using pop-ups are very low – rarely rising above 1% from visitors that are not existing customers.

Update: What I find most irritating for a user experience, is when the pop-up loads on the landing page. That is, you visit the site for the first time – you only have a vague idea of what the site is about (brand recognition is low), you have not read any of the content before and are not even sure if this is the place to be – yet on that very first page a pop-up survey appears and interrupts what you are attempting to do…

Even sites that should know better do this (e.g. www.emetrics.org)!

How to increase survey participation rates and get better responses from the correct target group without hurting your brand

Here are some alternative ways at soliciting feedback from your visitors without the use of an interuppting pop-up. The key is to get as many engaged visitors to take part in your survey as possible – a higher participation increases the likelihood of obtaining a statistically relevant outcome.

As a guide, if you have 10,000 visitors, you need survey results from 370 visitors to be 95% confident in your results to a level of + or – 5%. That’s a participation rate of 3.7%

Travel example Retail example How Google do it
BBC Social network This site…

The rationale of these examples is to solicit feedback from visitors at both ends of the user-experience spectrum. Why? Because those visitors that are unhappy with their experience are always more likely to provide their views. This leads to the “squeaky wheels” syndrome. That is, you have to be careful not to assume that everyone feels the same way just because you have not received as much positive feedback.

Check list for a best practise deployment of a survey

  1. Avoid pop-ups – allow users to provide their feedback using a standard text link or button
  2. Keep it simple – use transparent language for requesting participation
  3. Add your participation request close to the call to action or user event you wish to gain feedback on
  4. Keep participation anonymous – or keep personal information optional
  5. Avoid random participation – provide all your visitors the choice to send their feedback. If people want to talk with you, they will!
  6. Deploy continuously – as with web analytics data, a continuous and ongoing survey system provides you with much greater insight than running a survey for a single campaign or set time period.
  7. Integrate survey results with web analytics click stream data (the subject of a future post )

What are your thoughts on unsolicited pop-up surveys and feedback forms – good or bad for the user experience?

Share Button

Comments

  1. Refused4Reason says:

    Can anyone post the names, home addresses, and personal phone numbers of the people who create the ForeSee Results pop ups? I think many people would like to tell them directly how they feel about the pop up attacks.

  2. Refused4Reason says:

    I was considering investing in Kellogg because of their ad campaign for hunger-fighting pruducts such as Special K Protien cereal and a new Popcorn Chips.

    When visiting the Kellogg website http://www.specialk.com for the first time, and starting to read about these new products, the web page went dim and I got a “ForeSee Results” pop up attack. I closed I.E. and did not return to the web site. Responding to any pop up is too risky in damage to the computer and personal information. I thought I’d like to let people know how much damage ForeSee Results is doing to the web site and the brand and the company.

    At the moment, I’m researching to see if Kellogg is so stupid to kick their web site visitors in the face, or if an enemy of Kellogg’s http://www.specialk.com is causing the ForeSee Pop ups. I find that it looks like Kellogg doesn’t think about their customers or potential investors, so I don’t expect Kellogg to be a good investment. Some companies just don’t get it.

  3. Agree re popups…

    To keep this thread up to date: iPerceptions recently launched a new passive feedback system. As part of the launch process they published some useful data on response rates — which are low, as you might expect.

    I’ve written more about that and provided some comparison data of response rates to an embedded survey (a very different thing, so not comparing apples with apples):

    http://www.cxfocus.com/index.php/ecommerce-usability/customers-site/

    Tim

    • @Tim L_B Thanks for updating. In my experience I have found response rates much lower when requesting feedback “passively”. Shoving a survey in someones face will get you more responses, but it comes at a high costs – that is, a very poor user experience for the majority of people. Often this flies completely under the radar – silently killing your brand…!

  4. Tony says:

    And four years later, nothing has changed.

    Those annoying pop-up surveys are still around, and just as annoying as ever.

    Anyone considering using them should realize that MANY of us who encounter them have one of three responses:
    1) leave the site
    2) complain in the survey about annoying popup surveys
    3) provide false information

    Now, how does that help you to learn ANYTHING about your customers?

  5. asian store online says:

    I tried kissinsights and still had annoyed comments due to its use

  6. For the sake of completeness…

    The new version of the 4Q survey system now offers a minimalist invitation which rises up from the bottom of the screen like Kissinsights.

    http://www.iperceptions.com/solutions/4q/

    They also now offer tagging and categorisation of comments via their interface.

  7. paul says:

    yes, I tried kissinsights and still had annoyed comments due to its use.
    the hellobar isn’t converting as much as I thought it would. only 3 click throughs out of 1000 visits. and no one filled out the survey (google forms)
    the site is http://kingrichardsfaire.net if you’re interested

  8. Hi Brian, I’ve only got Kissinsights on a tiny test site, so I can’t claim much experience of it. My main reason for liking it was the way they encourage a minimalist to the questions.

    The invitation can be configured to only show itself if a variety of conditions are met, such as having viewed n many pages or been on a particular page for longer than a time threshold.

    They also provide a hack for triggering the popping of the invitation only when someone clicks on a specific link, so you could use that to come up with your own invitation system which required an explicit request. That last one might provide the answer. It takes more than clicking an option in their configuration system of course.

  9. Tim: I would be interested in your thoughts on KissInsights. When I tried it, it still used invasive “pop-up” technology. Can that be disabled?

  10. This thread contains a lot of thoughtful discussion and resources, so I thought I’d take the opportunity of this recent comment update with add a couple of new links to relevant systems:

    http://www.kissinsights.com/

    This is worth a look as it combines the more discrete approach to the survey invitations (like Kampyle) with a minimalist approach to the questions themselves.

    For those who are interested in the “Net Promoter” approach, I would also recommend taking a look at

    http://www.recommendi.com/

    which has an even more minimalist approach — just the ‘likely to recommend’ score and a free text comment box.

    Tim

  11. paul says:

    I was reading Web Analytics 2.0 which was published around the time of this post, and introduces 4Q, which is a popup survey for voice of customer.
    I nearly implemented it on my site, but then thought about how annoying they are and that I had Adblock plus installed.
    So instead I’m using the hellobar and a google form survey instead.

  12. Anon says:

    I found this blog post because I’m sick of looking at ForeSee Result popup and googled for a way to stop it.

    What vitrol greed-riddled oilyness over popup benefits, and how its all legal, and how Joe & Jane Average want popups. The truth is if only we (average people) had some way to equally annoy everyone working for (and paid by) popup companies with frustration, annoyance, and regularity, we’d waste as much of your time as you’ve wasted ours.

    At least some comments are made by people (Like John and Brian Clifton) in touch with average computer users.

    I think Google found me a “ForeSee Results Killer”
    Apparently “Ad Block Plus” for Mozilla Firefox can block a ” */foresee-trigger.js ” …

  13. John: lol, your sentiment is common among many web users. I have never met a user who is not annoyed by unsolicited pop-ups – and I ask many!

    Thank you for expressing it in writing. I feel this sentiment is the hidden part of the iceberg because the vast majority of people cannot be bothered to contact a web site owner about their poor user experience…

  14. John says:

    Larry Freed, you and Foresee Results can go to H*ll! As an angry user who sees your popups everywhere, I consider them intrusive spam and nothing else. They popup seconds after I enter a site, while I’m in the middle of browsing, while I’m doing something important like entering my credit card information! Is that doing things right or scientifically? Seems more like you’re pissing off half of your audience. Those who ACTUALLY take the survey get exposed to a way too long list that often contains very prying and intrusive questions. And at the very end, what to do people get? No coupons or drawings for prizes. Just solicitations for more surveys and mailing lists! As rule, if I see your popups, I intentionally give FALSE information. I encourage all the people who are sick and tired of survey popups to do the same!

  15. It would seem that consumers of today have built up a resistance to traditional forms of selling and advertising. The more you sell the more people resist it. Magnetic sponsoring offers a really big leverage for any marketer by tapping into a familiar concept called attraction marketing. Integrating magnetic sponsoring into your internet network marketing business will allow you to develop personal relationships with existing clients and become a person of value, so that they have the underlying feeling of attraction and want to come back for more. This authority networker method can attract an endless flow of free MLM leads and MLM signups so you can build your autopilot downline fast. Your prospects come to you, instead of you chasing after them.

  16. Rob says:

    With no previous VOC data to go on we just deployed 4Q. Will we do it as an on-going collection method? That remains to be seen. But, we also had not heard of kampyle.

    Maybe, we’ll try that next after we reach statistical certainty with 4Q.

  17. Tim Leighton-Boyce says:

    Stephane, you have inadvertently prompted me to make another observation about the perils of such VOC systems. I just clicked through to your site. As I type this, the relevant Firefox window (Title is now from your site) still tells me that it is ‘connected to cf.kampyle.com’ while displaying the previous page.

    It looks to me as if a delay in retrieving the survey code from Kampyle has halted the rendering of the page. I’ve seen this happen with things like Thawte seals and learnt to keep such code way down at the end of the page, as with tracking code etc.

    But if I am right (and I may not be, of course) then this is a ghastly lesson in how such systems can wreck customer experience.

    The information and insight is hugely important. I can’t overstate that. I’ve learnt so much and made so many fixes on the basis of such information. But the mechanisms for getting it still need to be refined. For the time being I have to leave requests for the information until much later in the visit. I’d love to know what those low-page-view people were trying to do, but I dare not ask. Yet.

  18. Very good post Brian. I always try to balance between my desire to know more about my visitors, especially with a startup and beta product like WASP, vs my own experience of being annoyed by pop-up/overlays ads and VOC requests…

    But you were kind enough to leave a comment when prompted to do so on http://wasp.immeria.net stating how you disliked unsolicited VOC requests. I took a few minutes to write a post at http://blog.immeria.net/2008/09/voc-balance-your-desire-to-know-and.html

    That being said, I’m becoming an advocate of Kampyle (and I see Eran is reading your blog too!). I find it a very good choice, being easier/friendlier than a SurveyMonkey and providing more in depth than something like 4Q.

  19. Eran: Great insight. I would say the biggest brand to benefit from such VoC feedback so far is Dell. Angus Cormie from Dell gave an excellent overview of this at this years’ London eMetrics summit, entitled “Listening skills at Dell”.

    The 2005 Dell Hell story of Jeff Jarvis is legendary. His comments, and subsequently other user’s comments on his buzzmachine.com blog, undermined the company and its credibility in the eyes of millions who read, or more commonly heard about the reliability and support problems on the grape vine.

    But Dell listened and responded with a huge investment in the three models you describe. Direct2Dell.com for example was their first effort of many.

    I know Starbucks and Harley Davidson invest heavily in VoC – do you have any other examples?

    Aman: I really like the Ideas Management of Yahoo (http://suggestions.yahoo.com). Looks like Eran could be a good contact for your questions.

  20. Aman Sandhu says:

    Hi Brian,

    Great post, while working at a large web portal, and being responsible for their User Generated Feedback program, the two things I found most difficult were

    1. Dealing with channel owners and their priorities (Read generic feedback forms vs. channel(site section) specific form.

    2.Volume of feedback -Contextual Analysis(We did develop an in-house text analysis tool, which analyzed free form text and categorized it into theme based categories with a positive, negative or neutral sentiment assigned to each theme.

    I’d like to hear from you on any best practices around the above two.

    Also, worth mentioning here as Eran pointed out was Idea management (check out http://suggestions.yahoo.com)

    Thanks
    Aman

  21. Eran Savir says:

    Brian,
    Thanks for the very interesting post about VOC. There are few different methodologies available to get the VOC: Feedback Form, Surveys and Idea Management. Each methodology serves a different need. Generally speaking I would say that:

    1. Feedback Form – if build right, placed all over the site, and have good analyze and management tools to support it – will provide you with high quality (i.e. specific comments) and actionable data, allowing you to read and manage feedback and also get back to the users. The feedback will be accessible to you only, allowing you judge it and decide what to do and how. This data can also be integrated into Web Analytics allowing you to understand both the ‘What’ and the ‘Why’.

    2. Surveys – usually you’ll get high level understanding of what’s going on. There are the problems that you mentioned in your post as for how to initiate the survey, but they will allow you to ask your users for many questions you’re interested to hear answers for (e.g. how did you hear about my site). Well, there is also the obvious downside for asking many questions.

    3. Idea Management – Have users suggest ideas, report problems etc., vote for other users’ ideas and have it publicly available to all. This can be very helpful to reduce your support efforts because some people will see that, e.g. in case of problem, its already a known one. It might also do good for some businesses looking to use their users publicly to vote for their next development steps. This solution can in some cases though damage your brand, in case lots of users have complaints, and others see it.

    Based on our experience, adding a prominent feedback button (or link) with a quick and easy-to-fill feedback form will cause the users to provide not only high quality actionable feedback but also to provide variety of opinions. Many users provide amazing ideas and compliments as a feedback and a huge number choose to provide their real email because they want to follow up on their feedback.

    Eran Savir
    Co-Founder and VP Business Development at Kampyle (www.kampyle.com)

  22. Thijs: “Don’t you think that a survey that presents itself at the start of the visit (and is presented “from the brand” with logo and domain) which asks permission to survey after the visit is different from the examples you’re showing?”

    – No, I don’t think this is different… the question is not how and when to display a pop-up to your visitors. That is working around what the visitor wants, instead of listening to their needs. The default for good visitor experience, is not to interrupt them unless they request “assistance” in the first place i.e. click on a feedback link.

    Mitch: I couldn’t agree more. Feedback from dissatisfied customers is just as valuable as those that are satisfied. The positive feedback allows you to reward your teams that are performing well and encourage more of the same, while negative feedback are your opportunities to improve. The key is for the survey not to skew results by turning a good or even average visitor experience into a bad one with “interruption marketing” (I really like that phrase! It does what is says on the tin…)

    Tim: I echo your experience and fears. I would only consider a pop-up if a visitor abandons a shopping cart/booking process. However, the caveat is that any web site that requires an “add to cart” action in order to ascertain the full price (e.g. travel) needs special consideration.

  23. Larry Freed says:

    Brian,
    Great post! And I even agree with most of it. A couple of points. VOC (surveys) can be used for creating engagement with your visitors or provide accurate measurement. If using it for creating engagement, an opt-in survey works fine. But, you will not get a representative sample and therefore you will not get an accurate measurement. If you are looking for an accurate measurement (incredibly valuable!) you must have a random sampling. Which means you must randomly intercept users and invite them to take a survey. We have had great success with this approach, averaging nearly 10% response rate over millions of surveys a year. With no giveways, prizes, contests, etc. Pop-ups, done well, with an invite first, is a great strategy.

    You are right on with the value of attitudinal data and when merged with behavioral data they are a potent combination.

    Another important point is voice of customer data is only as good as the methodology. And the methodology includes not only the survey approach, the questions, but also how the data is analyzed. While it sounds simple, most satisfaction measurement programs fail. Why? Their measurements are not accurate, precise, reliable and most of all, not predictive of the future. A satisfied customer will be a long term, loyal (and profitable) customer — when measured correctly. If you can rely on a proven (and I mean PROVEN) methodology, such as the American Customer Satisfaction Index (and the UK National Customer Satisfaction Index) your data will not only be accurate, precise and reliable, but also predictive and actionable.

    -Larry Freed
    CEO ForeSee Results (www.ForeSeeResults.com)

  24. Tim Leighton-Boyce says:

    Thisj: my experience of running 4Q on a test site is that I do receive comments from visitors who have been annoyed. Things along the lines of “I WAS trying to do [whatever] until this survey…”

    Other people have mentioned similar comments in the support forum. Although it’s true that the invitation can be dismissed and true that it does explain how the survey is intended to be ignored until the visit is complete, nevertheless there is evidence of people being offended. And these comments are from people to who took the trouble to voice their complaint. So it’s reasonable to assume that there will others, possibly more who did not even stay that long.

    That’s why I have not yet dared use the system on a real e-commerce site even though I am a very strong advocate of ‘voice of the customer’ and would even go so far as to choose ‘VOC’ over traditional clickstream analysis if I was only allowed one tool for analytics!

  25. Mitch says:

    Sometimes hearing the customers opinion from time to time might be troublesome to hear but if companies want to grow they need to take the constructive criticism and build off of it…

  26. Hi Brian,

    Just read your book (which was excellent) over the holiday in Italy and now stumble more or less by coincidence on this post. It’s like when you buy a new car you all of a sudden see that make everywhere 😉

    Interesting post as surveying VOC is indeed something I’m recommending to all my clients.

    Don’t you think that a survey that presents itself at the start of the visit (and is presented “from the brand” with logo and domain) which asks permission to survey after the visit is different from the examples you’re showing? The former builds a picture of the overall experience while I suspect that the latter I suspect is often used to comment on specific sections or pages?

    Obviously I am referrring to the 4Q tool Avinash created with iPerceptions which does indeed use a lightbox (modern day popup) to present itself.

    I’m not convinced it is so bad to be honest – people will see the lightbox when they come to the site. If they do not want to participate they simply click it away. Will they be annoyed? Maybe some, slightly, but so much that it really hurts the brand? You could be right but I am not sure. I hope to collect more evidence now that 4Q is localized into other languages then English.

  27. Tim: That’s a good point. The only place where a pop-up is justified is on cart abandonment i.e when the visitor leaves the purchase process. They have decided not to buy from you, so why not!

    The problem though, is that many sites require you to “add to cart”, or start the booking process in order to get the full price (travel is a good example of that situation). The result is that visitors regularly abandom the purchase process for perfectly valid reasons, and probably go on to purchase at a later date. So a pop-up survey for them is just as irritating.

    Esstentially the point of my post was that with all the positive buzz around voc, I would hate to see the web analytics world take such a giant step back by using pop-ups for deployment…

  28. Nick: yes exactly – you need to avoid the squeaky wheels syndrome. The suggsted examples I give are intended to be agnostic. That is, whether you are happy or not with the experience, the call to provide feedback is intuitively visible and close to the call to action (or pain point!)

    Dixon: Yes small world indeed! I love that part of Spain. I hear we are on the same panel at Stockholm SMX – see you there 😉

    Yes, the BBC site is there as an example of encouraging participation. It shouldn’t matter what the final intent is. The point is to generate engagement with your visitors.

    Of course the BBC is in a unique position and isn’t really comparable with any business web site, but I wanted to use it because in terms of participation level, I don’t think there is a site out there that matches it. Even comparing to Facebook and Myspace.

  29. Tim Leighton-Boyce says:

    I share your reservations about the pop-ups which is why we’ve never deployed them on any of the sites I work on.

    But I am a huge fan of voice of customer data. That’s why I was very interested to read this post.

    And that’s why I DO try, for example to get surveys embedded in as many e-commerce confirmation pages I can. Of course, that’s a very specific section of the visitors, but the comments from the free-text box are a real goldmine nevertheless.

    Kampyle have an interesting approach to invitation, but a little too cutesy (and the survey examples I’ve seen with the smileys even more so) for some markets I think.

  30. Dixon Jones says:

    Hiya Brian. Missed you by minutes in Murcia!

    The BBC example isn’t so much to generate feedback as it is to build up user generated content as far as I understand.

    I like the “inline” idea of asking for feedback – ideally at the point at which a person is interacting woth the site anyway… IE on a web form adding:

    + Answer two questions while you fill in this form to get free delivery

    With the link revealing the questions on the page (not pop up).

  31. Nick says:

    The problem I find with voice of customer is that you are not able to select a representative group. Is there any evidence that the types of people who fill in online survey forms don’t just represent a minority profile with different preferences to those that don’t fill in the survey?

Leave a Reply

Your email address will not be published. Required fields are marked *


6 − 4 =

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

© Brian Clifton 2015
Best practice privacy statement