I am studying customer satisfaction in a large hierarchical organization. I plan to administer a voluntary survey to customers across the organization, and need to address non-response in my analysis.
- How many customers there are (population size)
- Some demographic/market segment data for all customers
- The products/services they used
- The account manager and department that handles their account
- 50% response rate expected
I want to draw conclusions about departments, account managers and products. It'd also be nice to be able to break things out by customer demographics, but that's secondary.
The naive approach would assume that non-response is random and treat the data as a simple random sample of the population. My concern is that non-response is probably not completely random.
So here are my questions:
- Is the 50% response rate sufficiently high that the naive approach isn't terrible?
- If not, is resampling or post-stratification a sufficient remedy?
- If not, does it make sense to try to model the sampling probability mechanism using the known customer demographic information?
- If none of these are sufficient, would I be better off designing a smaller stratified sample and aggressively pursuing non-responders?
- Does SurveyMonkey ignore the fact that you get a non-random sample?
- Recommend references on survey sample weighting
- Ways to overcome small number of survey responses
- Are benchmarking and post-stratification equivalent?
EDIT 4/22/2014: Reviewing @Steve Samuels response and additional independent research, I think what I'm dealing with is a census with non-response. The population (all current customers) is sufficiently small and well-known to serve as the sample frame. Thus by definition we have a census. It's actually less feasible to address a random sub-population than the whole population due to the platform we're using to deliver the questionnaire.
My plan moving forward is to execute the census, then study the difference between the responding & non-responding populations in a variety of ways. Moving forward, follow-up efforts will be adjusted to address specific problems found in the non-response analysis.
Some good resources for this problem include:
- Analysis of Customer Surveys: with applications using R (Kenett and Salini, 2011) has a good chapter on sample design and continues with a solid overview of survey analysis techniques, all with a focus on customer satisfaction surveys.
- The causes and consequences of response rates in surveys by the news media and government contractor survey research firms (Holbrook, Krosnick, and Pfent, 2007) does a meta-analysis of 114 phone surveys conducted by professional survey firms and "challenges the assumptions that response rates are a key indicator of survey data quality and that efforts to increase response rates will necessarily be worth the expense"
- Survey response rate levels and trends in organizational research (Bruch and Holtom, 2008) provides benchmarks for response rates in organizational surveys (50% is typical) and guidelines for responsibly reporting response statistics.