Questionnaire Design Risks: The Challenge of Staying Focused

(part of Chapter 1: Know Your Goals)

 

One of the main reasons MR projects fail is due to lack of an agenda, or the inability to stay focused on an originally projected design. As more individuals become involved with a project, they bring with them the risk of transforming a once precise questionnaire into an excessively long one with illogical flow.

The most important aspects to keep in mind when designing a questionnaire are fit, flow and length. Without these, the questionnaire becomes a burden on the respondents, which directly affects project success. Read on to find out how to avoid the pitfalls of questionnaire design.

 

Very risky business

One of the most common reasons MR projects fail is because there is no specific agenda. Perhaps it started with one, but then the goals were diluted as different decision-makers started to contribute to the instrument design.

It is a painfully common scenario: You start with a very precise MR study topic. Perhaps it’s a project to test potential messages for a new advertising campaign. However, once word gets out that you’re planning to do a custom MR project, it seems everybody wants to add “just” two or three questions to the questionnaire: “As long as you’re talking to our customers, can you please ask them about distribution channels?” or, “Can you please find out if they’ve seen this recent ad we ran?”

Before you know it, a very precise, focused questionnaire that can be done with excellence has now become a questionnaire with an illogical flow that doesn’t make any sense to the people who are taking the survey. Now, instead of being a nice 10-minute questionnaire, it’s a 30-minute monster. Even worse, the people screened for the primary objective may be unqualified to answer these tangential questions—so the resulting data end up weak—and now all of those unwelcome survey vampires are unhappy internal clients!

Alas, it is time to be the survey cop. Someone has to define, protect, and uphold the agreed upon project priorities. Does that mean you should just say no to these requests? Of course not. Sometimes the requestor can be shown that the answers to their questions already exist. In their excitement to add on to a new project, perhaps they have forgotten to look for existing information.

A little guidance about where the answers may possibly exist could solve the conflict. And if truly no readily available answers exist, it may simply be time to scope a new project to address their specific needs. Once you select the agency for your primary study, you may find some economies of scale by having them bid on this new project as well.

 

Why Is This So Risky?

If you decide to add questions to your instrument that are not specifically aligned with your primary goals, you have to keep in mind three things: fit, flow, and length.

 

Fit. Are the people you’re going to recruit for the primary goals also the right people to answer these new questions? If not, you might need to revamp the original sampling plan, which can drive up costs and introduce delays.

• Most studies require very specific samples (the people who will be qualified to take your questionnaire or participate in your focus group).
  
  For example, let’s say the primary goal of this project is to measure awareness of your brand. You are likely interested in measuring awareness only among people who actually buy your product category (you might not want to measure awareness of your frozen pizza brand among people who never buy frozen pizza). You’re going after a very specific audience.


• If you want to add questions outside of the original scope, you have to decide if the same population can answer the additional questions. Perhaps, instead, they would be best answered by a differently qualified group. Maybe now you would have to change your recruiting parameters to include both people who already buy your product category and those who do not.
  
  In the pizza example, perhaps your company is now thinking about extending its brand into frozen hamburgers; so now you want to find out if your brand has permission to enter that category. It may be valid to test your brand’s permissions among people who do and do not currently buy frozen pizza.


• Unless you have a large budget for extra sampling, it’s probably not going to be a good idea. If a project requires dramatically different lines of questioning for two separately qualified groups, it is almost always best to simply conduct two separate studies.

 

Flow. Another thing to consider before adding extra questions to an instrument is this: Do the extra questions in any way bias participants’ responses to the rest of the questionnaire? That is, are we going to be introducing a topic or a concept that might somehow change how people would respond to subsequent questions? It is important to be careful about this. Will new questions need to be sequenced in a way that is awkward to respondents to avoid biasing responses?

• Of course, you also want to avoid asking questions that are too cognitively difficult or intrusive—you don’t want to cause dropouts over secondary objectives.

 

Length. For surveys, shorter durations mean higher response and completion rates. Unnecessarily long or complicated questionnaires can drive up costs and drive down data quality—and do so very quickly.

 

This is an excerpt from the book, "How to Hire & Manage Market Research Agencies," which is available on Amazon. Published by Research Rockstar LLC. Copyright © by Kathryn Korostoff. All rights reserved.