July 16, 2007
Consumer Survey Overkill
By Tom Ryan
According to Jupiter Research, the value of customers’ feedback from surveys will diminish over time due to excess surveying and a “self-selecting surveying audience.” The research firm also found that consumer surveys tend to be overly positive.
In a study, Jupiter found sizeable differences in survey-participation rates for customers following good and bad service experiences. Jupiter analyzed consumers who contacted customer service about an online purchase during the previous six months as part of its November 2006 Jupiter Research/Ipsos Insight survey. Of those who opted to participate in a survey, 59 percent said they were providing feedback on a good interaction, while only 40 percent said they were discussing experiences that were unsatisfactory.
The study appears to contradict the conventional thought that displeased consumers are more likely than happy ones to participate in consumer surveys.
Zach McGeary, lead analyst on the report, told destinationcrm.com that this phenomenon may be because consumers have grown more accustomed and open to surveys. As it has become known that companies are relying more heavily on customer feedback to gauge and improve service, consumers are “overwhelmingly responding by providing feedback across a variety of channels.”
But the positively skewed information collected from surveys will likely lead companies to believe they are providing a more satisfying service experience than they actually are.
“Service executives will be challenged to look beyond feedback surveys for informing their service optimization initiatives going forward,” Mr. McGeary said.
While Mr. McGeary sees consumer surveys as an effective way “to capture the voice of the customer,” he said companies must not rely on customer feedback alone.
“Our mandate is that companies leverage customer feedback in a more tactical manner, driving insight down to the customer level, allowing companies to affect customer experiences in real time,” he said. Currently, satisfaction and feedback data is largely leveraged for macro-level initiatives that provide little actionable insight beyond branding and marketing initiatives, according to Mr. McGeary.
“To inform more tactical initiatives that would have a more direct impact on the customer experience, companies must also bring customer profile/history data as well as web site analytics data,” Mr. McGeary said. “The ultimate use for customer feedback should be to understand why consumers are satisfied or dissatisfied, not simply whether they are satisfied or not.”
Discussion Questions: Do you think the value of consumer surveys is diminishing through excessive surveying? What factors should be considered when analyzing consumer surveys? What other methods – particularly cost-effective ones – might be required to gain a more accurate reading of the consumer?
BrainTrust
Recent Discussions
Discussion Questions
Poll
90% of surveys are filled out just to get free offers.
This makes them totally meaningless – particularly meaningless for companies that score their employees as a zero because they did not score 100%.
Many of the questions that can cause a zero rating have nothing to do with the employee’s performance.
For example, “we are very sorry but we do not have any whole bellied clams today due to the recent weather conditions.”
Result is that survey is 90% – everything was perfect except item was out of stock. The result is employee is rated a zero, and is actually chastised for something completely out of their control.
My gut tells me it’s definitely having an adverse affect. Every time I buy something online, stay in a hotel or fly somewhere, I’ve got an email asking me all about my experience, the weather in Vermont and my grandmother’s shoe size. Why don’t companies put together a group of questions, and rotate them around to their lists so that each potential respondee has to answer only one question, not 27? And they lie. They tell you it’ll take only a minute, but it takes 15. They don’t offer you anything meaningful anymore as an incentive. If a hotel chain offered me $20 off my next stay, I might consider it. But when I’m trying to barrel thru 100 emails and get my work done, no way. I might respond only if something is really terrific, or really horrible. Otherwise, who has the time?
The true value of consumer surveys depends on the truth, purpose, and level of skill in which the survey is conducted. A few variables to look for in measuring the value and its validity:
1. Who conducted the survey? How do they get paid? Was the survey conducted by a professional company in an objective manner? Or, was the survey conducted or assisted by the surveying retailer, company, or brand? Was the survey assisted by a public relations or advertising firm?
2. What was the objective of the survey? Was the goal to find the truth about what consumers think about a retailer, product, or its competition, or was the goal really to confirm positive things that can be used for advertising, or to confirm someone’s performance?
3. How were the questions of the survey formulated? Were responses left open for true feedback, or did the deck of cards get stacked so that most respondents would not be as critical as they might actually feel?
4. How were the results interpreted, and by whom? Did the interpreting party have proper disposition to study and learn the results for real, or would there be either positive or negative consequences depending on the outcome?
5. Is the selection of survey-type the right format to serve the most meaningful objectives for the task at hand?
Surveys are as good as intention, formulation, and interpretation. If executed objectively and resourcefully, surveys will always have a greater value than other means, because they look not only into the past, but also the present and into the future to measure and predict attitudes and trends.
Some colleagues and I were talking about this last week in relation to the degradation of service and if we felt that anyone truly cared anymore. Often, consumers feel that firms are “going through the motions” to appear that they truly care…and then we don’t experience a tangible manifestation of this caring (think the airline industry!)
As a former brand planner, we often found that people would skew their answers towards what they thought we wanted to hear. In reality, their behavior in-environment was much different. Maybe this is coming into play here? One would wonder if this is why user-generated reviews are growing in popularity and believability. It’s also why Ethnography is so critical to incorporate into research efforts to understand consumers’ TRUE perceptions and experiences.
Of course we are being over-surveyed. We need to dump our traditional tools (or at least give them a rest) and adopt more tools from the social sciences, especially anthropology and ethnography.
I believe that customer surveys are not a good gauge of what the consumer is thinking/feeling/doing. Especially if the survey takes place anytime after 48 hours of when the transaction occurs. Of the data that is eventually compiled, I think 2 most important questions on any survey would be “Were you satisfied with the product/service?” and ‘Will you be returning in the future?” Those are really the 2 most important pieces of data that any retailer needs to know. If the answer is NO to either questions, then the surveyor needs to investigate further and get details as to why NO was chosen.
A better way to gauge customer perceptions is to establish a ‘CustomerCare’ department with email and telephone access. And information about CustomerCare would be posted near the exits/registers of a bricks and mortar store as well as printed on receipts or on the final transaction page (the page the customer prints out) for online transactions. Customers can offer complaints or compliments almost immediately while the transaction is still fresh on their minds if they know they have unfettered access to some sort of customer service department. This is the raw data that retailers need analyze how they are perceived by the consumer.
A very legitimate issue, but we have not seen any deterioration in research value over time. We always try to validate responses with information that is known through other data sources (e.g. Point-of-Sale). Using that criteria, we still get great validation with online survey research, even though there is the risk of over-use of respondents.
The benefit of consumer research is imperative and valuable! But,like all information and data gathering means, there are some improper and overused vehicles. So the response to the question for this article is yes.
This is why you need constant engagement with your target audience, and not the occasional or ‘shotgun’ communications research means, on an ongoing basis. The consumer is always the starting and ending point of any business. Hmmmmm
Not every company can support ethnography, yet consumer behavior tracking is important to most marketers we work with, either on the retail or consumer goods side of the coin. Survey research can be a very useful tool, particularly if results are trended over time to help you spot shifts. Even better if the research results can be viewed multiple ways via segments. Smart marketers can then decide if they need to use additional research methods to dig deeper on key issues where shifts are occuring. Looking at consumer survey data for Adults 18+ is pretty much useless if you are marketing to men 18-24, for example. On the other hands, if you can sub-segment men multiple ways, comparing the segments to the base and to female sub-segments, you can track key differences and spot shifts in behavior pretty easily, and then set up a strategy to understand what is driving the shifts in behavior.
We have noted for the last 20 years that every business organization we have ever encountered mis-uses surveys. Let’s look at a sample question: “How would you rate the knowledge of the sales staff you used?” There are two ways to find out this information–one is by asking the customer, one is by asking the sales staff. Here’s another: “About how long did you wait on hold on the phone?” You can ask customers, but this information is available from within your own company. Here is another, trickier one: “How would you rate the quality of the food of your meal?” This is information that you can get by doing a focus-study, creating metrics, and then measuring the food yourself.
What this boils down to is this: businesses use surveys to get information that they could easily and less expensively get themselves, that is more reliable, and doesn’t burden customers. Point one.
Point two: our data does not show any significant improvement in the quality of goods or services in organizations that use surveys, no matter what type of survey they use, even after five years. This may offend some readers or seem incredible. I’ll share data with anyone who wants to see it. A good example of this is a very well known credit-card processing company, who has surveyed the blazes out of their merchants for years, yet you wait 30 minutes on the phone for a service rep, the service rep has no clue how to help you, you are not allowed to speak to a supervisor, and they NEVER reply to emails.
The problem isn’t surveys use/misuse, the problem is organizations have a very difficult time facing and processing information that puts them in a poor light. This information is blocked, distorted, and just not uncovered. Organizations who want to really rise above their competition have an incredibly easy way to do so, but they suffer from this same problem, so it seems incredibly difficult.
The findings of Jupiter Research are not surprising. According to a study by Utah State University, 96% of dissatisfied customers go to a competitor without stating the reason for their defection. While Consumer Surveys serve a distinct purpose, most retailers conduct surveys from people who made a purchase. Usually, shoppers who make NO purchase also don’t answer the survey. Thus the results are further skewed because the retailer might be offering inadequate service and losing sales.
When used properly, consumer surveys tell what the consumer feels is important. Mystery shopping measures, objectively, whether or not the service promised by the retailer or deemed important by the consumer is actually being delivered.
Consumer surveys do not aid in the retraining of an associate’s behavior…mystery shopping does, as it measures whether or not the desired behavior is being exhibited.
Consumer surveys are important, but they are not the only tool. Surveys are one step in the improvement of overall customer service.
Like any other management tool, surveys can be used well or poorly. The key is repeated consistency. Even if there is a response bias (toward the unhappy customers or the happy customers), that bias won’t matter much if the same survey is used many times, over the course of several years, because trends start to reveal themselves. If the survey is only done intermittently, with different questions each time, discerning any trend would be difficult. Translation: look at the direction, not the details. If the survey is done monthly for 12 months in a row, and satisfaction seems to be rising, the trend is your friend. And since surveys can be done as samples, only a small percentage of shoppers need to be approached.
Self selection and who is being interviewed is a large factor at determining how valid surveys are. The consumer retail landscape is littered with failures from poor surveys (remember new Coke, or Microsoft’s “Bob”)? Surveys are more art than science and folks need to recognize this any time that they are considering large investments, leveraging a brand or even extending a product line. The key to success is to remain grounded, have a clear, transparent communication system in place for the marketing feedback and to just use common sense. Like Mark Twain said “There are lies, damn lies and statistics.”
Absolutely, consumers are receiving too many surveys–especially too many surveys that ask for similar information and provide no value to the consumer. As a result, consumers don’t see a good reason for completing the surveys. If you think about it, your time is valuable and when a request comes across your desk for your time you may well ask the question “is it worth doing this?” or “do I have to do this?” Consumers ask the same question when receiving lots of similar surveys.
The value to the company is marginal with traditional satisfaction surveys. Service research says that satisfied consumers are vulnerable if competitors offer service with an added value. So you do a survey that indicates your consumers are satisfied–it says nothing about whether they are loyal or whether they would leave for a better product or service–so what do you really know if your customers are satisfied? To retain consumers, research indicates that consumers have to be delighted–every time!! Traditional surveys generally do not reveal that information.
“Do you think the value of consumer surveys is diminishing through excessive surveying? Take the Poll.” (SMILE:)
As noted, surveys can be good or bad, instructive or self-fulfilling ego-boosters (more than useless): there’s nothing new here, and professionalism is necessary to get it right…as with anything else in business (or life). What is new or at least more widespread (I think) with the rise of the internet is the emergence of a relatively small group of “professional” survey takers: this is problematic to the extent that the group is seen as representative–and may indeed begin as such–but isn’t.
As a long time disciple of Paco Underhill, I place little to no value on surveys because I believe there is more to learn by analyzing what customers do as opposed to what they say. Knowing what they are doing involves knowing what is going on in your own stores and those of others. This is not speculative knowledge or data but real and real time. It doesn’t happen in too many retail organizations because they have retail execs who think they are too important for the “floor”…which is why so many organizations get worse as they get big and so many retail executives manage to the past instead of the present (because they no longer really know what is going on).
Require all management to spend quality time every week in stores with their help, their merchandise, and their customers and they will have more understanding of customers and store needs than the most effective survey could ever give. And, by the way, a fancy “walk through” and talking only to the store manager is not “quality time” in a store.
The objective of consumer surveys is to reveal the truth. But truth is what we think it is (or what we want it to be) at any given moment of time; prevalent emotion; or on our sociological dependence. Ryan’s suggestion would seem to make more sense.
The value of transactional customer sat research collected immediately via IVR or online is incalculable! Not only does it track something that is important but it also creates a living bible that enforces a culture throughout the organization of what matters. Synovate also builds programs that allow a dissatisfied customer to be contacted within one day from the manager of the service center to intervene before the dissatisfaction leads to losing a customer. Woe unto the manager who does not follow up as his manager can view metrics to make sure that occurs. Next subject!
When I have worked with companies that have surveyed their customers over a period of years (on customer experience with company, experience with competitors, how and why they acquire the product, demographics, etc.) the results have changed very little, which either indicates that they are fairly accurate or consistently skewed in the same direction.
All surveys are subject to response bias. After all, the first answer is always, “Yes, I’ll participate.”
I strongly suspect that Americans are being over-surveyed so much that the survey method is no longer transparent to them. Web-based panels and other online mechanisms have contributed to this. Simple quantitative surveys are easy to defeat by respondents who want to accumulate incentive points. Online anonymity only makes this easier.
So I agree that fewer surveys and more qualitative study, including ethnographic and observational approaches, would help companies discover the reality behind consumer choices. Don’t just ask consumers, “What would you do?”; watch what they actually do. Don’t just ask about satisfaction; monitor action.
The biggest problem with surveys and polls has ALWAYS been in the administration of them.
There is a scientific approach, dealing with statistical analysis, that should be applied to any meaningful information gathering effort.
However, it’s not easy and it’s costly to get statistically significant samples from a controlled population without bias.
Therefore, as discussed, most companies cut corners and create self serving questionnaires that offer no value to customers, and usually, no meaningful, actionable data other than what was probably already known.
All the political polls you hear about in the media are prime examples of horrible statistical standards being used to “create” a result that props up someone’s agenda. I have yet to meet anyone who has actually participated in a published political poll and I’ve heard many others state the same.
Unfortunately, the public, (which makes up the marketplace) is so flooded with this type of thing that commercial polling seems to have taken on the same shape. Emphasizing this further is the growth of online social/info/pop culture sites like sodahead.com where you can create your own “survey” on any subject imaginable. (i.e.–hotdogs…relish or sauerkraut?)
Yes, surveys are overdone. And true, quality market research is nearly dead.
It is a poor carpenter that blames his tools.
Consumer surveys themselves aren’t becoming less effective, but I believe they are being increasingly misused. As the demand for consumer feedback increases across all industries, employees with little or no training in marketing research are being called upon to design consumer surveys and interpret their results. Anyone can ask a question, and someone will eventually answer it, but that’s a risky approach to gathering market intelligence, particularly if important decisions hang in the balance. The value of marketing research that is designed, executed, and interpreted correctly is immeasurable. Conversely, poorly designed marketing research, with biases introduced during data collection, and irresponsible interpretation is at best useless, and at worst misleading.
I do not believe that consumers, in general, are being excessively surveyed. That said, I would not be surprised in the least if there were pockets of consumers who were being over-surveyed by these well-intentioned but misguided company representatives. Like anyone engaged in a decidedly one-sided conversation, these consumers will eventually lose interest and their feedback will become unproductive.
I believe all data is good data when viewed through the appropriate filter. When assessing the value of survey data it is important to consider the source (who conducted the research?), the sample (who, exactly, was interviewed?), and the data collection methodology (how and when were the data collected?). A well-designed market and brand intelligence program will use any and all methods of data collection across a wide variety of consumer touch points. The risk in such a program comes from the over-reliance and blind faith in a single method or measure.
Marketplace dynamics, particularly the attitudes and opinions that drive consumer behavior, are inherently complex and may change rapidly. To position your company for success, you will need to use all sources of information at your disposal. Resist the temptation to oversimplify the results or draw grandiose conclusions from limited data points. Rather, look for trends, patterns, and consistency. Remember, H.L. Mencken said “for every complex problem, there is a solution that is simple, neat, and wrong!”
It is a poor carpenter that blames his tools.
Consumer surveys themselves aren’t becoming less effective, but I believe they are being increasingly misused. As the demand for consumer feedback increases across all industries, employees with little or no training in marketing research are being called upon to design consumer surveys and interpret their results. Anyone can ask a question, and someone will eventually answer it, but that’s a risky approach to gathering market intelligence, particularly if important decisions hang in the balance. The value of marketing research that is designed, executed, and interpreted correctly is immeasurable. Conversely, poorly designed marketing research, with biases introduced during data collection, and irresponsible interpretation is at best useless, and at worst misleading.
I do not believe that consumers, in general, are being excessively surveyed. That said, I would not be surprised in the least if there were pockets of consumers who were being over-surveyed by these well-intentioned but misguided company representatives. Like anyone engaged in a decidedly one-sided conversation, these consumers will eventually lose interest and their feedback will become unproductive.
I believe all data is good data when viewed through the appropriate filter. When assessing the value of survey data it is important to consider the source (who conducted the research?), the sample (who, exactly, was interviewed?), and the data collection methodology (how and when were the data collected?). A well-designed market and brand intelligence program will use any and all methods of data collection across a wide variety of consumer touch points. The risk in such a program comes from the over-reliance and blind faith in a single method or measure.
Marketplace dynamics, particularly the attitudes and opinions that drive consumer behavior, are inherently complex and may change rapidly. To position your company for success, you will need to use all sources of information at your disposal. Resist the temptation to oversimplify the results or draw grandiose conclusions from limited data points. Rather, look for trends, patterns, and consistency. Remember, H.L. Mencken said “for every complex problem, there is a solution that is simple, neat, and wrong!”
Statistical sampling has not changed at all. Getting people to donate their time to think about someone else’s problem has gotten very hard. Electronically, someone could code a program to ask millions of questions for millions of years. But no one is going to pay any attention.
There is some irony in my (our) answer(s) though, as we’re thoughtfully participating in a poll right now. RetailWire’s technique is brilliant.
Consumer surveys can be an integral part of “capturing the voice of the customer,” provided the survey can analyze and quantify 100 percent of open-ended customer surveys. Many surveys check scores but often don’t have a way to capture and analyze the written text that consumers include in their responses. Text analytics is a crucial piece to a successful interpretation of a survey. Many companies still process this textual information manually, and as a result, only gather a small percentage of their customers’ written comments. Gathering and analyzing the open-ended text portion of surveys via text analytics technology is pivotal because it explains “why” customers are making their decisions, versus learning simply the “what” they are doing. (For example it would explain “why” a customer took back their product, versus simply showing that they returned their product.)
The dynamic nature of the Web has enabled consumers the ability to provide feedback across a wide variety of channels and their comments (good or bad) often reach a large audience. Companies seeking to provide the best customer service and to therefore gain a competitive edge must combine their customer survey information with other unstructured data, such as online product reviews, call center notes, CRM text fields, blogs and other Web content. By incorporating all of this unstructured data with a company’s internal structured data, such point-of-sale information typically found in databases and spreadsheets, businesses can gain a true 360 degree view of their customers’ needs, emotions and desires. Consumer surveys will remain an invaluable piece to gaining customer insight, the trick is to be able to analyze all customer comments and feedback from both structured and unstructured information sources in a timely and effective manner and turn that insight into action.
More Discussions
Has Same-Day Delivery Become an Expectation Across Retail?
Are expectations around retail delivery speed in flux?OpenAI Names the Infrastructure Standard for Agentic Commerce
Will OpenAI or Google (or another entity) set the gold standard for agentic commerce?Is Five Below Poised For Even Greater Growth?
Five Below is hitting serious growth numbers. Can this trend continue?
90% of surveys are filled out just to get free offers.
This makes them totally meaningless – particularly meaningless for companies that score their employees as a zero because they did not score 100%.
Many of the questions that can cause a zero rating have nothing to do with the employee’s performance.
For example, “we are very sorry but we do not have any whole bellied clams today due to the recent weather conditions.”
Result is that survey is 90% – everything was perfect except item was out of stock. The result is employee is rated a zero, and is actually chastised for something completely out of their control.
My gut tells me it’s definitely having an adverse affect. Every time I buy something online, stay in a hotel or fly somewhere, I’ve got an email asking me all about my experience, the weather in Vermont and my grandmother’s shoe size. Why don’t companies put together a group of questions, and rotate them around to their lists so that each potential respondee has to answer only one question, not 27? And they lie. They tell you it’ll take only a minute, but it takes 15. They don’t offer you anything meaningful anymore as an incentive. If a hotel chain offered me $20 off my next stay, I might consider it. But when I’m trying to barrel thru 100 emails and get my work done, no way. I might respond only if something is really terrific, or really horrible. Otherwise, who has the time?
The true value of consumer surveys depends on the truth, purpose, and level of skill in which the survey is conducted. A few variables to look for in measuring the value and its validity:
1. Who conducted the survey? How do they get paid? Was the survey conducted by a professional company in an objective manner? Or, was the survey conducted or assisted by the surveying retailer, company, or brand? Was the survey assisted by a public relations or advertising firm?
2. What was the objective of the survey? Was the goal to find the truth about what consumers think about a retailer, product, or its competition, or was the goal really to confirm positive things that can be used for advertising, or to confirm someone’s performance?
3. How were the questions of the survey formulated? Were responses left open for true feedback, or did the deck of cards get stacked so that most respondents would not be as critical as they might actually feel?
4. How were the results interpreted, and by whom? Did the interpreting party have proper disposition to study and learn the results for real, or would there be either positive or negative consequences depending on the outcome?
5. Is the selection of survey-type the right format to serve the most meaningful objectives for the task at hand?
Surveys are as good as intention, formulation, and interpretation. If executed objectively and resourcefully, surveys will always have a greater value than other means, because they look not only into the past, but also the present and into the future to measure and predict attitudes and trends.
Some colleagues and I were talking about this last week in relation to the degradation of service and if we felt that anyone truly cared anymore. Often, consumers feel that firms are “going through the motions” to appear that they truly care…and then we don’t experience a tangible manifestation of this caring (think the airline industry!)
As a former brand planner, we often found that people would skew their answers towards what they thought we wanted to hear. In reality, their behavior in-environment was much different. Maybe this is coming into play here? One would wonder if this is why user-generated reviews are growing in popularity and believability. It’s also why Ethnography is so critical to incorporate into research efforts to understand consumers’ TRUE perceptions and experiences.
Of course we are being over-surveyed. We need to dump our traditional tools (or at least give them a rest) and adopt more tools from the social sciences, especially anthropology and ethnography.
I believe that customer surveys are not a good gauge of what the consumer is thinking/feeling/doing. Especially if the survey takes place anytime after 48 hours of when the transaction occurs. Of the data that is eventually compiled, I think 2 most important questions on any survey would be “Were you satisfied with the product/service?” and ‘Will you be returning in the future?” Those are really the 2 most important pieces of data that any retailer needs to know. If the answer is NO to either questions, then the surveyor needs to investigate further and get details as to why NO was chosen.
A better way to gauge customer perceptions is to establish a ‘CustomerCare’ department with email and telephone access. And information about CustomerCare would be posted near the exits/registers of a bricks and mortar store as well as printed on receipts or on the final transaction page (the page the customer prints out) for online transactions. Customers can offer complaints or compliments almost immediately while the transaction is still fresh on their minds if they know they have unfettered access to some sort of customer service department. This is the raw data that retailers need analyze how they are perceived by the consumer.
A very legitimate issue, but we have not seen any deterioration in research value over time. We always try to validate responses with information that is known through other data sources (e.g. Point-of-Sale). Using that criteria, we still get great validation with online survey research, even though there is the risk of over-use of respondents.
The benefit of consumer research is imperative and valuable! But,like all information and data gathering means, there are some improper and overused vehicles. So the response to the question for this article is yes.
This is why you need constant engagement with your target audience, and not the occasional or ‘shotgun’ communications research means, on an ongoing basis. The consumer is always the starting and ending point of any business. Hmmmmm
Not every company can support ethnography, yet consumer behavior tracking is important to most marketers we work with, either on the retail or consumer goods side of the coin. Survey research can be a very useful tool, particularly if results are trended over time to help you spot shifts. Even better if the research results can be viewed multiple ways via segments. Smart marketers can then decide if they need to use additional research methods to dig deeper on key issues where shifts are occuring. Looking at consumer survey data for Adults 18+ is pretty much useless if you are marketing to men 18-24, for example. On the other hands, if you can sub-segment men multiple ways, comparing the segments to the base and to female sub-segments, you can track key differences and spot shifts in behavior pretty easily, and then set up a strategy to understand what is driving the shifts in behavior.
We have noted for the last 20 years that every business organization we have ever encountered mis-uses surveys. Let’s look at a sample question: “How would you rate the knowledge of the sales staff you used?” There are two ways to find out this information–one is by asking the customer, one is by asking the sales staff. Here’s another: “About how long did you wait on hold on the phone?” You can ask customers, but this information is available from within your own company. Here is another, trickier one: “How would you rate the quality of the food of your meal?” This is information that you can get by doing a focus-study, creating metrics, and then measuring the food yourself.
What this boils down to is this: businesses use surveys to get information that they could easily and less expensively get themselves, that is more reliable, and doesn’t burden customers. Point one.
Point two: our data does not show any significant improvement in the quality of goods or services in organizations that use surveys, no matter what type of survey they use, even after five years. This may offend some readers or seem incredible. I’ll share data with anyone who wants to see it. A good example of this is a very well known credit-card processing company, who has surveyed the blazes out of their merchants for years, yet you wait 30 minutes on the phone for a service rep, the service rep has no clue how to help you, you are not allowed to speak to a supervisor, and they NEVER reply to emails.
The problem isn’t surveys use/misuse, the problem is organizations have a very difficult time facing and processing information that puts them in a poor light. This information is blocked, distorted, and just not uncovered. Organizations who want to really rise above their competition have an incredibly easy way to do so, but they suffer from this same problem, so it seems incredibly difficult.
The findings of Jupiter Research are not surprising. According to a study by Utah State University, 96% of dissatisfied customers go to a competitor without stating the reason for their defection. While Consumer Surveys serve a distinct purpose, most retailers conduct surveys from people who made a purchase. Usually, shoppers who make NO purchase also don’t answer the survey. Thus the results are further skewed because the retailer might be offering inadequate service and losing sales.
When used properly, consumer surveys tell what the consumer feels is important. Mystery shopping measures, objectively, whether or not the service promised by the retailer or deemed important by the consumer is actually being delivered.
Consumer surveys do not aid in the retraining of an associate’s behavior…mystery shopping does, as it measures whether or not the desired behavior is being exhibited.
Consumer surveys are important, but they are not the only tool. Surveys are one step in the improvement of overall customer service.
Like any other management tool, surveys can be used well or poorly. The key is repeated consistency. Even if there is a response bias (toward the unhappy customers or the happy customers), that bias won’t matter much if the same survey is used many times, over the course of several years, because trends start to reveal themselves. If the survey is only done intermittently, with different questions each time, discerning any trend would be difficult. Translation: look at the direction, not the details. If the survey is done monthly for 12 months in a row, and satisfaction seems to be rising, the trend is your friend. And since surveys can be done as samples, only a small percentage of shoppers need to be approached.
Self selection and who is being interviewed is a large factor at determining how valid surveys are. The consumer retail landscape is littered with failures from poor surveys (remember new Coke, or Microsoft’s “Bob”)? Surveys are more art than science and folks need to recognize this any time that they are considering large investments, leveraging a brand or even extending a product line. The key to success is to remain grounded, have a clear, transparent communication system in place for the marketing feedback and to just use common sense. Like Mark Twain said “There are lies, damn lies and statistics.”
Absolutely, consumers are receiving too many surveys–especially too many surveys that ask for similar information and provide no value to the consumer. As a result, consumers don’t see a good reason for completing the surveys. If you think about it, your time is valuable and when a request comes across your desk for your time you may well ask the question “is it worth doing this?” or “do I have to do this?” Consumers ask the same question when receiving lots of similar surveys.
The value to the company is marginal with traditional satisfaction surveys. Service research says that satisfied consumers are vulnerable if competitors offer service with an added value. So you do a survey that indicates your consumers are satisfied–it says nothing about whether they are loyal or whether they would leave for a better product or service–so what do you really know if your customers are satisfied? To retain consumers, research indicates that consumers have to be delighted–every time!! Traditional surveys generally do not reveal that information.
“Do you think the value of consumer surveys is diminishing through excessive surveying? Take the Poll.” (SMILE:)
As noted, surveys can be good or bad, instructive or self-fulfilling ego-boosters (more than useless): there’s nothing new here, and professionalism is necessary to get it right…as with anything else in business (or life). What is new or at least more widespread (I think) with the rise of the internet is the emergence of a relatively small group of “professional” survey takers: this is problematic to the extent that the group is seen as representative–and may indeed begin as such–but isn’t.
As a long time disciple of Paco Underhill, I place little to no value on surveys because I believe there is more to learn by analyzing what customers do as opposed to what they say. Knowing what they are doing involves knowing what is going on in your own stores and those of others. This is not speculative knowledge or data but real and real time. It doesn’t happen in too many retail organizations because they have retail execs who think they are too important for the “floor”…which is why so many organizations get worse as they get big and so many retail executives manage to the past instead of the present (because they no longer really know what is going on).
Require all management to spend quality time every week in stores with their help, their merchandise, and their customers and they will have more understanding of customers and store needs than the most effective survey could ever give. And, by the way, a fancy “walk through” and talking only to the store manager is not “quality time” in a store.
The objective of consumer surveys is to reveal the truth. But truth is what we think it is (or what we want it to be) at any given moment of time; prevalent emotion; or on our sociological dependence. Ryan’s suggestion would seem to make more sense.
The value of transactional customer sat research collected immediately via IVR or online is incalculable! Not only does it track something that is important but it also creates a living bible that enforces a culture throughout the organization of what matters. Synovate also builds programs that allow a dissatisfied customer to be contacted within one day from the manager of the service center to intervene before the dissatisfaction leads to losing a customer. Woe unto the manager who does not follow up as his manager can view metrics to make sure that occurs. Next subject!
When I have worked with companies that have surveyed their customers over a period of years (on customer experience with company, experience with competitors, how and why they acquire the product, demographics, etc.) the results have changed very little, which either indicates that they are fairly accurate or consistently skewed in the same direction.
All surveys are subject to response bias. After all, the first answer is always, “Yes, I’ll participate.”
I strongly suspect that Americans are being over-surveyed so much that the survey method is no longer transparent to them. Web-based panels and other online mechanisms have contributed to this. Simple quantitative surveys are easy to defeat by respondents who want to accumulate incentive points. Online anonymity only makes this easier.
So I agree that fewer surveys and more qualitative study, including ethnographic and observational approaches, would help companies discover the reality behind consumer choices. Don’t just ask consumers, “What would you do?”; watch what they actually do. Don’t just ask about satisfaction; monitor action.
The biggest problem with surveys and polls has ALWAYS been in the administration of them.
There is a scientific approach, dealing with statistical analysis, that should be applied to any meaningful information gathering effort.
However, it’s not easy and it’s costly to get statistically significant samples from a controlled population without bias.
Therefore, as discussed, most companies cut corners and create self serving questionnaires that offer no value to customers, and usually, no meaningful, actionable data other than what was probably already known.
All the political polls you hear about in the media are prime examples of horrible statistical standards being used to “create” a result that props up someone’s agenda. I have yet to meet anyone who has actually participated in a published political poll and I’ve heard many others state the same.
Unfortunately, the public, (which makes up the marketplace) is so flooded with this type of thing that commercial polling seems to have taken on the same shape. Emphasizing this further is the growth of online social/info/pop culture sites like sodahead.com where you can create your own “survey” on any subject imaginable. (i.e.–hotdogs…relish or sauerkraut?)
Yes, surveys are overdone. And true, quality market research is nearly dead.
It is a poor carpenter that blames his tools.
Consumer surveys themselves aren’t becoming less effective, but I believe they are being increasingly misused. As the demand for consumer feedback increases across all industries, employees with little or no training in marketing research are being called upon to design consumer surveys and interpret their results. Anyone can ask a question, and someone will eventually answer it, but that’s a risky approach to gathering market intelligence, particularly if important decisions hang in the balance. The value of marketing research that is designed, executed, and interpreted correctly is immeasurable. Conversely, poorly designed marketing research, with biases introduced during data collection, and irresponsible interpretation is at best useless, and at worst misleading.
I do not believe that consumers, in general, are being excessively surveyed. That said, I would not be surprised in the least if there were pockets of consumers who were being over-surveyed by these well-intentioned but misguided company representatives. Like anyone engaged in a decidedly one-sided conversation, these consumers will eventually lose interest and their feedback will become unproductive.
I believe all data is good data when viewed through the appropriate filter. When assessing the value of survey data it is important to consider the source (who conducted the research?), the sample (who, exactly, was interviewed?), and the data collection methodology (how and when were the data collected?). A well-designed market and brand intelligence program will use any and all methods of data collection across a wide variety of consumer touch points. The risk in such a program comes from the over-reliance and blind faith in a single method or measure.
Marketplace dynamics, particularly the attitudes and opinions that drive consumer behavior, are inherently complex and may change rapidly. To position your company for success, you will need to use all sources of information at your disposal. Resist the temptation to oversimplify the results or draw grandiose conclusions from limited data points. Rather, look for trends, patterns, and consistency. Remember, H.L. Mencken said “for every complex problem, there is a solution that is simple, neat, and wrong!”
It is a poor carpenter that blames his tools.
Consumer surveys themselves aren’t becoming less effective, but I believe they are being increasingly misused. As the demand for consumer feedback increases across all industries, employees with little or no training in marketing research are being called upon to design consumer surveys and interpret their results. Anyone can ask a question, and someone will eventually answer it, but that’s a risky approach to gathering market intelligence, particularly if important decisions hang in the balance. The value of marketing research that is designed, executed, and interpreted correctly is immeasurable. Conversely, poorly designed marketing research, with biases introduced during data collection, and irresponsible interpretation is at best useless, and at worst misleading.
I do not believe that consumers, in general, are being excessively surveyed. That said, I would not be surprised in the least if there were pockets of consumers who were being over-surveyed by these well-intentioned but misguided company representatives. Like anyone engaged in a decidedly one-sided conversation, these consumers will eventually lose interest and their feedback will become unproductive.
I believe all data is good data when viewed through the appropriate filter. When assessing the value of survey data it is important to consider the source (who conducted the research?), the sample (who, exactly, was interviewed?), and the data collection methodology (how and when were the data collected?). A well-designed market and brand intelligence program will use any and all methods of data collection across a wide variety of consumer touch points. The risk in such a program comes from the over-reliance and blind faith in a single method or measure.
Marketplace dynamics, particularly the attitudes and opinions that drive consumer behavior, are inherently complex and may change rapidly. To position your company for success, you will need to use all sources of information at your disposal. Resist the temptation to oversimplify the results or draw grandiose conclusions from limited data points. Rather, look for trends, patterns, and consistency. Remember, H.L. Mencken said “for every complex problem, there is a solution that is simple, neat, and wrong!”
Statistical sampling has not changed at all. Getting people to donate their time to think about someone else’s problem has gotten very hard. Electronically, someone could code a program to ask millions of questions for millions of years. But no one is going to pay any attention.
There is some irony in my (our) answer(s) though, as we’re thoughtfully participating in a poll right now. RetailWire’s technique is brilliant.
Consumer surveys can be an integral part of “capturing the voice of the customer,” provided the survey can analyze and quantify 100 percent of open-ended customer surveys. Many surveys check scores but often don’t have a way to capture and analyze the written text that consumers include in their responses. Text analytics is a crucial piece to a successful interpretation of a survey. Many companies still process this textual information manually, and as a result, only gather a small percentage of their customers’ written comments. Gathering and analyzing the open-ended text portion of surveys via text analytics technology is pivotal because it explains “why” customers are making their decisions, versus learning simply the “what” they are doing. (For example it would explain “why” a customer took back their product, versus simply showing that they returned their product.)
The dynamic nature of the Web has enabled consumers the ability to provide feedback across a wide variety of channels and their comments (good or bad) often reach a large audience. Companies seeking to provide the best customer service and to therefore gain a competitive edge must combine their customer survey information with other unstructured data, such as online product reviews, call center notes, CRM text fields, blogs and other Web content. By incorporating all of this unstructured data with a company’s internal structured data, such point-of-sale information typically found in databases and spreadsheets, businesses can gain a true 360 degree view of their customers’ needs, emotions and desires. Consumer surveys will remain an invaluable piece to gaining customer insight, the trick is to be able to analyze all customer comments and feedback from both structured and unstructured information sources in a timely and effective manner and turn that insight into action.