Data is not Information: Surveys Need Context

From the Cheese Reporter. Volume 137, No. 21 - Friday, November 16, 2012

​And Not All Information is Understanding

I don’t know about you, but it seems like every time I go to a website these days, an invitation to fill out a survey pops up. Now I have a kind of rebellious nature, and find these things an invasion of my privacy and an interruption. I rarely text and when busy, let the calls go to voicemail.

I understand why everyone is doing it, with all the hoopla in the business media about listening to your customers, even from me. But knowing something about statistics as it relates to market research, I am afraid for the kinds of decisions people are going to make based on a poorly designed substitute for genuine dialogue with their customers.

It reminds me of those silly boxes they had in every company I ever worked in, called the suggestion box. Garbage in, garbage out, and most of what went in that box had nothing to do with useful suggestions, and rarely got read.

Nothing against the darnfangled things, it is just that it’s not the outward form that ensures the suggestions will be listened to, and in companies that really want and listen to suggestions, you don’t usually have to pop them in a box.

The desire to get input from other people must start from the heart. But even this is not a guarantee of useful, actionable information. Suggestion boxes, and pop-up surveys provide little useful information, just a bunch of noise, as they provide very little to no context with which to be able to understand.

So, 35 people who had nothing better to do with their lives than spend time on your website answering questions when they should be working said x, y and z. How do you know if what they are saying is true for other people, never mind the majority of people you sell to?

What if no, and you change your product, in the process losing those who buy even more? Data can only be understood, and turned into information, in context. Dr. Shewhart’s first rule for the presentation of data is: data should always be presented in a way that preserves the evidence in the data, to ensure the accuracy of predictions made from these data.

What the data is trying to tell us exists only in its context. If you don’t know, cannot get someone to clearly and simply describe the context the data represents, you simply can’t trust it. Or more simply, are you dealing with apples, or are you dealing with oranges, or both? How can you possibly know this from a bunch of people who just happened to show up?

Now, I know what you’re thinking, why did this guy write his last column about how to write good survey questions only come back in the next column and tell me I can’t trust the information I get from surveys?

Good question!

It’s not so much that you can't trust the information in homemade surveys, but that you have to be careful. If you’ve targeted the survey, and you know how to target a survey statistically to ensure the right sample of people has been chosen to give you useful information about what you want to know, great! But most of us don’t have a hoot of an idea how to do that. If you’re like the rest of us, you have to cast a cold eye on any information you get from your survey, your staff, even the handful of people you have actually talked to, and have to apply the law of three: if you hear it from one source don’t believe, if he heard from two sources it may have some truth but be careful, if you hear it from three different sources, it’s probably true, but go to where the action is, where people interact with your product and see with your own eyes. In homegrown market research, and in truth, in most “professionally” done research, even when three sources agree, you can’t really believe it until you test it in reality.

We are bombarded by statistics all the time. It’s far easier for the media to throw statistics at us, than it is to do the hard work of real journalism. The only thing worse than all those political polls that tell us what we are supposed to think if we want to remain popular, is all those cute little business reporters on the nightly news who every day keep telling us “stuff” about why the stock market went up or went down, when the truth is simply over time the stock market goes up and goes down within limits. They haven’t got a clue.

Heck, just to talk with a few people a day takes hours out of my day. When they say, “the general feel of the market is that the market went down because of the fear of x,y,z” why do we take them seriously? It is gossip, not news, and projecting reasons on things that happen day to day most often simply because of natural variation. So it’s simply not enough to rely on any one way of getting information. You have to see through their eyes for yourself, and get to talk to them directly, and take information from surveys you do yourself, simple ones, and take information from professional surveys; with an open mind, think about them in your inner core, and test them in reality to see if they really make sense, continuously, because things change.

If the only people you get information from are those people already buying from you who visit your website, and have the time and inclination to respond, how will you ever grow your business? Wouldn’t it be good to know what people who aren’t currently buying your product think as well?

The one thing we never see, after they show us polls, and offer up their absurd explanations, is what really happened. Nor can we know what it all means, because they never share the context. As long as we keep tuning in they keep throwing this stuff at us. But can you afford to run business the same way?

If living with uncertainty; moving step by step and testing reality is not acceptable to you, then pony up for a professional, or keep tossing coins and flying blind. Understanding how to define the context; to choose the right target population to match that context; and how to create the sample to ensure a high confidence in the results is a science: if you have the need and the means pay someone professional to do a thorough study, and explain its real impact to you. If not, like the rest of us, settle for what you can get, carefully.

About the Author

Dan Strongin works with medium to small companies, helping them master the art and science of managing.

Leave a Reply 0 comments