7
$\begingroup$

What justifies the usage of a variable for post-stratification?

I am working with a constituent survey of a non-profit's constituent with 2500 responses out of a much larger sample and even larger population. I have many variables about the target population, which are all active constituents. In literature I've read, it's common to use demographic variables (age, gender, and race, for example), but in my experience with this data, demographics have relatively high data quality errors and weak correlation to non-response error, while behavioral data (for example, donation history) are recorded reliably and correlate better to non-response.

I assume demographics are common because many surveys try to get a nationally representative sample, and the government publishes demographic information for this population.

Because I have them, is there anything wrong with using the behavioral variables instead of, or in addition to, the demographics? Is there a practical empirical method to choose variables?

If the suggestion is to use behavioral variables in addition to demographics, how would I detect or prevent overfitting when raking weights with many variables?

$\endgroup$
  • $\begingroup$ What kind of data quality errors do you expect to see with demographics? In my experience, those are the most reliable. Most people know their age with a great degree of precision, for example. You are correct, on the other hand, that many of them tend to vary less than some behavioral metrics with different levels of effort to reduce non-response. But, most surveys tend to under-sample young people and hispanics. You would need to know what the "true" distribution is in order to weight them correctly, of course. US Census doesn't apply to organization memberships. $\endgroup$ – Jonathan Oct 24 '12 at 17:11
  • $\begingroup$ @Jonathan: Especially in the population (but probably also in the survey) we have data entry errors for age. In the population age was collected from various sources including an overlay from various data vendors, and using seed records, I have confirmed an error rate of about 10%. In the population we have lots of missing values for age (about 40%), and we have a house holding problem: one constituent (donor) is multiple people (example: husband, wife, and kids), some with significantly different ages. Also we don't know the "primary" for the household, so I'd have to guess which age to use. $\endgroup$ – Andrew Oct 24 '12 at 21:19
  • $\begingroup$ yeah, that seems like a lot of data quality issues :) $\endgroup$ – Jonathan Oct 25 '12 at 18:43
4
$\begingroup$

It is an interesting twist to find a case where the demographic data will be seen as less reliable as the behavioral data. There isn't much good advice on how to select the calibration variables, other than they should correlate with both the (non-)response process and the variables of interest. The reasoning behind the widespread use of demographic variables for calibration is that age, education, race and gender affect pretty much everything in any social science. You can make a very simple case with your data, however, by modeling the probability of response as a function of all the variables that you think about including -- a propensity model, if you like. If you can demonstrate that donations are more significant in your model than age, nobody would have the grounds to object to your use of the former in calibration.

The question of how much calibration is enough has not been addressed much, either. I can think about this conceptually as a trade-off between improving the accuracy (which, for a given response variable $y$ and a set of calibration variables $\bf x$, is the variance of the residuals $e_i = y_i - {\bf x}_i' {\bf b}$) and the increase in the variability of the weights, and hence the design effect $1+{\rm CV}^2$. As you add predictors that decrease in their strength, the precision gains are diminishing; the CV though will continue increasing, so at some point, arguably, the two curves will meet, giving you the right number of calibration variables. That's just an idea, but may be I should write a paper about it :)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.