Variables for post-stratification weights?

Question

What justifies the usage of a variable for post-stratification?

I am working with a constituent survey of a non-profit's constituent with 2500 responses out of a much larger sample and even larger population. I have many variables about the target population, which are all active constituents. In literature I've read, it's common to use demographic variables (age, gender, and race, for example), but in my experience with this data, demographics have relatively high data quality errors and weak correlation to non-response error, while behavioral data (for example, donation history) are recorded reliably and correlate better to non-response.

I assume demographics are common because many surveys try to get a nationally representative sample, and the government publishes demographic information for this population.

Because I have them, is there anything wrong with using the behavioral variables instead of, or in addition to, the demographics? Is there a practical empirical method to choose variables?

If the suggestion is to use behavioral variables in addition to demographics, how would I detect or prevent overfitting when raking weights with many variables?

Answer 1

It is an interesting twist to find a case where the demographic data will be seen as less reliable as the behavioral data. There isn't much good advice on how to select the calibration variables, other than they should correlate with both the (non-)response process and the variables of interest. The reasoning behind the widespread use of demographic variables for calibration is that age, education, race and gender affect pretty much everything in any social science. You can make a very simple case with your data, however, by modeling the probability of response as a function of all the variables that you think about including -- a propensity model, if you like. If you can demonstrate that donations are more significant in your model than age, nobody would have the grounds to object to your use of the former in calibration.

The question of how much calibration is enough has not been addressed much, either. I can think about this conceptually as a trade-off between improving the accuracy (which, for a given response variable $y$ and a set of calibration variables $\bf x$, is the variance of the residuals $e_i = y_i - {\bf x}_i' {\bf b}$) and the increase in the variability of the weights, and hence the design effect $1+{\rm CV}^2$. As you add predictors that decrease in their strength, the precision gains are diminishing; the CV though will continue increasing, so at some point, arguably, the two curves will meet, giving you the right number of calibration variables. That's just an idea, but may be I should write a paper about it :)