Using a sample of paid survey respondents to bias correct lower response rate among larger non-paid sample

Question

I'm running a tracker survey on a website that has a low response rate of about 2%. The survey is not incentivized but the website traffic is large enough in volume I can always meet my sample target of 5,000 each month.

I have pretty good data on the population of website users so I can weight my sample accordingly so in a statistical sense it looks the same as the population. However, there is still a large amount of selection bias in terms of who decides to participate and who doesn't (98% of sampled website visitors choose not to respond!).

For example, if my sample has 30% female respondents and 70% male respondents but I know that my population is evenly split (50/50), then I can upweight the 30% of female respondents and downweight the 70% of male respondents until in a statistical sense my sample looks the same as the population. The issue here is that i'm making a big assumption that the delta of 20% of female respondents I would have obtained in a random sample (and a 100% response rate) would have responded the same way to the questions in my survey as the 30% who did respond (where the response rate is only 2%). I.e. we can weight on observables, but are doing nothing for unobservables.

So to try and account for this selection bias, I had an idea to run a single, one-off, paid version of the survey, which was heavily incentivized. Based on previous paid surveys we've run on the website, and the budget I have available, I think I can pay enough to get the response rate up to 40%(!). Still less than half, but 20x what i'd see otherwise. My thinking is I can then use this data to correct in some way for the bias that I see in the ongoing unpaid version.

But this raises some questions:

1. Paying people will introduce a new kind of selection bias, as we know from the literature that people who are likely to participate in a paid survey look different from people who participate in non-paid surveys, and more importantly, look different from a random sample of the population with a 100% response rate. We also know that paying people can actually change responses even if they would have participated in both the paid and unpaid versions. It still feels worth a shot though, as by significantly increasing the response rate we reduce the type of selection bias we observe in the non-paid version.
2. I'm unsure what the mechanics of the bias correction would look like. I believe the data could help me, but i don't know exactly what i'd have to do with it.

So, to my questions, does anyone have experience doing this kind of thing before? (i doubt i've come up with a totally new way of correcting for selection bias in surveys.) And does this sound like a good approach? Any big pitfalls I should be aware of, or is this an awful idea and I should abandon immediately?

Would welcome any thoughts!

Answer 1

The most important thing is to keep your eye on the ball: Your population of interest. That is the whole reason you're sampling. Incentives are good if you think the entire population will increase the number of responses - not just some of them. Let's step back a minute.

1. If you somehow know the M:F split is something like 50:50, then if you got M:F 70:30, it's not likely that occurred by chance. That's where the bias is. Look to correct that.
2. If an incentivized method existed that brought 70:30 all the way down to 50:50, then that incentivized method represents a population of Women and Extremely Greedy Men.
3. If you pee in the pool, you have a dirty pool.
4. Remember: Quality data is your only chance at meaningful results. Garbage in - garbage out.
5. If you focus on quality, the numbers will come (even if the error variance is higher than you like). If you focus on quantity, you'll get n. In fact, send me an incentivized survey.
6. Survey methods for maintaining quality while increasing sample size are in abundance, and worth your time: https://surveymethods.com/15-ways-to-increase-survey-response-rates/