# Variables for post-stratification weights?

What justifies the usage of a variable for post-stratification?

I am working with a constituent survey of a non-profit's constituent with 2500 responses out of a much larger sample and even larger population. I have many variables about the target population, which are all active constituents. In literature I've read, it's common to use demographic variables (age, gender, and race, for example), but in my experience with this data, demographics have relatively high data quality errors and weak correlation to non-response error, while behavioral data (for example, donation history) are recorded reliably and correlate better to non-response.

I assume demographics are common because many surveys try to get a nationally representative sample, and the government publishes demographic information for this population.

Because I have them, is there anything wrong with using the behavioral variables instead of, or in addition to, the demographics? Is there a practical empirical method to choose variables?

If the suggestion is to use behavioral variables in addition to demographics, how would I detect or prevent overfitting when raking weights with many variables?

• What kind of data quality errors do you expect to see with demographics? In my experience, those are the most reliable. Most people know their age with a great degree of precision, for example. You are correct, on the other hand, that many of them tend to vary less than some behavioral metrics with different levels of effort to reduce non-response. But, most surveys tend to under-sample young people and hispanics. You would need to know what the "true" distribution is in order to weight them correctly, of course. US Census doesn't apply to organization memberships. – Jonathan Oct 24 '12 at 17:11
• @Jonathan: Especially in the population (but probably also in the survey) we have data entry errors for age. In the population age was collected from various sources including an overlay from various data vendors, and using seed records, I have confirmed an error rate of about 10%. In the population we have lots of missing values for age (about 40%), and we have a house holding problem: one constituent (donor) is multiple people (example: husband, wife, and kids), some with significantly different ages. Also we don't know the "primary" for the household, so I'd have to guess which age to use. – Andrew Oct 24 '12 at 21:19
• yeah, that seems like a lot of data quality issues :) – Jonathan Oct 25 '12 at 18:43

The question of how much calibration is enough has not been addressed much, either. I can think about this conceptually as a trade-off between improving the accuracy (which, for a given response variable $y$ and a set of calibration variables $\bf x$, is the variance of the residuals $e_i = y_i - {\bf x}_i' {\bf b}$) and the increase in the variability of the weights, and hence the design effect $1+{\rm CV}^2$. As you add predictors that decrease in their strength, the precision gains are diminishing; the CV though will continue increasing, so at some point, arguably, the two curves will meet, giving you the right number of calibration variables. That's just an idea, but may be I should write a paper about it :)