# Using a sample of paid survey respondents to bias correct lower response rate among larger non-paid sample

I'm running a tracker survey on a website that has a low response rate of about 2%. The survey is not incentivized but the website traffic is large enough in volume I can always meet my sample target of 5,000 each month.

I have pretty good data on the population of website users so I can weight my sample accordingly so in a statistical sense it looks the same as the population. However, there is still a large amount of selection bias in terms of who decides to participate and who doesn't (98% of sampled website visitors choose not to respond!).

For example, if my sample has 30% female respondents and 70% male respondents but I know that my population is evenly split (50/50), then I can upweight the 30% of female respondents and downweight the 70% of male respondents until in a statistical sense my sample looks the same as the population. The issue here is that i'm making a big assumption that the delta of 20% of female respondents I would have obtained in a random sample (and a 100% response rate) would have responded the same way to the questions in my survey as the 30% who did respond (where the response rate is only 2%). I.e. we can weight on observables, but are doing nothing for unobservables.

So to try and account for this selection bias, I had an idea to run a single, one-off, paid version of the survey, which was heavily incentivized. Based on previous paid surveys we've run on the website, and the budget I have available, I think I can pay enough to get the response rate up to 40%(!). Still less than half, but 20x what i'd see otherwise. My thinking is I can then use this data to correct in some way for the bias that I see in the ongoing unpaid version.

But this raises some questions:

1. Paying people will introduce a new kind of selection bias, as we know from the literature that people who are likely to participate in a paid survey look different from people who participate in non-paid surveys, and more importantly, look different from a random sample of the population with a 100% response rate. We also know that paying people can actually change responses even if they would have participated in both the paid and unpaid versions. It still feels worth a shot though, as by significantly increasing the response rate we reduce the type of selection bias we observe in the non-paid version.
2. I'm unsure what the mechanics of the bias correction would look like. I believe the data could help me, but i don't know exactly what i'd have to do with it.

So, to my questions, does anyone have experience doing this kind of thing before? (i doubt i've come up with a totally new way of correcting for selection bias in surveys.) And does this sound like a good approach? Any big pitfalls I should be aware of, or is this an awful idea and I should abandon immediately?

Would welcome any thoughts!