Best Survey Practices

By: David Zirkle, PhD

Surveys are a great way of collecting information to support the decision-making process.  Surveys are a cost-effective approach to collect large amounts of information in a relatively short period of time.  As a result, healthcare organizations are using surveys to address a growing number of information needs.  However, designing, executing and analyzing a survey can be time-consuming.  And, if not done correctly, the process can result in misleading data, which is worse than no information at all.

Remember, for a survey to be effective, the information must be distributed to the right people, at the right time, in the right format. Completing a successful survey project requires multiple steps:

1. Establish the goals of the project — What you want to learn
2. Determine your sample — Whom you will interview
3. Choose interviewing methodology — How you will interview
4. Create your questionnaire — What you will ask
5. Pre-test the questionnaire — Test the questions
6. Conduct interviews — Ask the questions
7. Analyze the data — Crunch the numbers
8. Produce the report — Summarize the results

The following is a guide that will walk you through each step in the survey process.

To learn more about Barlow McCarthy and our survey capabilities, please contact us at info@barlowmccarthy.com.

Survey Planning and Creation Guide

Project Planning

1. Clearly define research objectives.
2. Review existing literature to see how topics have been addressed in the past.
3. Review with colleagues and stakeholders to validate objectives.
4. Determine if a survey is the best method to achieve objectives (e.g. are there better ways to collect the information, consider the cost vs. benefits, likelihood of success).
5. Determine target audience (e.g. seniors, healthcare decision-makers, population at large).
6. Identify sample “frame” or source of potential respondents (e.g. email list, phone numbers, mail addresses, patient list).
7. Determine sampling method to select potential respondents (e.g. random, stratified, convenience, other).
8. Identify best survey method to address objectives and reach target audience (e.g. mail, phone, web).
9. Determine sample size or quota needed to provide needed level of confidence in results.

Design Survey

1. Develop questions
   1. Use primarily close-ended questions to collect information.  Include open-ended questions when necessary. Are there too many open-ended questions?  If so, go back to #2 above.
   2. Determine appropriate scales for close-ended questions (e.g. yes/no, agreement scale, multiple-point rating scale, etc.).
   3. Ensure multiple-choice question responses are mutually exclusive (only one response option can be true) and exhaustive (all response options cover the full spectrum of possible answers).
   4. Include N/A, don’t know or not sure response options when appropriate – use when some respondents are not likely to be familiar with all of the survey topics.
   5. Avoid biased or leading questions (e.g. “Do you think our customer service is the best?”)
   6. Avoid acronyms, jargon, and slang that respondents may not be familiar with.
   7. Randomize multiple-choice options so the order presented to the respondent is not always the same.
   8. Remove redundant or unnecessary questions to avoid respondent fatigue and minimize survey length.  Always identify need-to-know vs. nice-to-know questions and remove latter if survey length is excessive.
2. Introduction
   1. Explain who is conducting the survey, the topics covered and why the information is important and/or how it will be used.
   2. Include estimate of time to complete the survey.
   3. Explain if survey provides confidentiality (only investigators can identify responses of individual participants) or anonymity (participants can’t be linked to their responses).
   4. Disclose if incentive is included for participation.
   5. Thank respondent for their time.
3. Format
   1. Start with easy questions first.  Place difficult or sensitive questions near the end of survey.
   2. Ask important questions early in the survey to ensure this information is collected.
   3. Consider the flow and question sequence.  As a rule, sequence questions from general to specific and place questions with similar content together.
   4. Use “skip logic” to avoid asking questions not relevant to the respondent and minimize survey length.
   5. Put demographic questions at end.
   6. Include an open question at the end of survey for additional comments and/or to address items not included (e.g. “please provide any other comments you have?”)
4. Field test survey with trial population to identify any weaknesses.  Review test data and make necessary revisions before distributing final survey.

Administer Survey

1. Distribute survey according to defined methodology.
2. Response rate:

1. Keep length to 10 minutes or less. On average, respondents can complete five close-ended questions per minute and two open-ended questions per minute.  Longer surveys can be conducted when necessary, but larger samples will be required to meet quotas due to lower response rates.
2. “Identifying” the sponsor will generally increase response rates since participants know who is conducting the survey.  However, “blinding” the survey is often necessary to ensure validity or anonymity of a client who does not want to be named (e.g. new product or service).
3. Incentives will generally increase response rates.

3. Send follow-up “reminders” to potential respondents to encourage participation,usually at one-week intervals until quota is met.
4. Monitor responses periodically to track response rate and ensure data is collected correctly.

Analyze Survey

1. Review and validate data before beginning analysis.
2. Code open-ended responses into mutually exclusive categories.
3. Tabulate data according to standards (e.g. segment responses by key demographic, socio-economic categories, etc.).  Note:  There are several software programs available to tabulate survey data including Microsoft Excel and Access.
4. Step back and review data to get the “big picture.”
5. Use frequency tables and/or graphs to summarize and display the survey data.
6. Use cross-tabs to compare sub-groups or different segments of interest.
7. Use filters to analyze all questions for a particular subset of responses (e.g. those that responded negatively to a particular question).

Report Findings

1. Consider the audience before deciding on a report format (e.g. CEO vs. program manager, etc.).
2. Microsoft PowerPoint is often used to present graphical reports are that are light on text and heavy on graphs and charts.  Word is used for more detailed reports (e.g. findings that require significant explanation and studied in great detail).
3. Avoid introducing personal opinions not supported by the data.
4. Consider the following report outline and modify as needed.
   1. Background and Objectives
   2. Executive Summary
   3. Survey/Research Population (including sampling methods)
   4. Research Methodology
   5. Data Summary (including tables and charts to display results)
   6. Key Conclusions
   7. Recommendations or Next Steps