Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My spouse frequently works with (expensive, hard to obtain) data samples; for example route information for commuting bicyclists collected using a smartphone app. More often than not, these samples suffer from some kind of known demographic over-representation that they'd like correct for various applications.

Mindful of Karl Roves' and friends "corrections" to "obvious" democrat oversampling for the Nov. 2012 election polls which led to rather embarrassingly incorrect predictions, is there any theoretically appropriate way of doing this?

I'm not even sure what to call what I'm looking for -- is this what in some places is called reject inference?

share|improve this question
1  
Try googling 'survey design weights'. I think those are what you want. – conjugateprior Nov 14 '12 at 21:52
    
In general "weighting" is what you're looking for... but in the example you give, weighting may be insufficient to the task. Weighting corrects for over- and under-representation, but it won't correct for relevant categories of people being totally excluded, for example because they don't own a smartphone. – Jonathan Nov 15 '12 at 20:08

This is the fundamental point of weighting a sample to population. You weight each individual in your sample based on known demographic features of the population such that the weights of each demographic group in the sample add up to population totals.

See any book on sampling theory and practice - no, it's not reject inference.

I recommend Thomas Lumley's survey package in R and the accompanying book on complex surveys. Even if you don't use R it is a great, clear introduction.

share|improve this answer
    
Many thanks. I do use R extensively, but will also check out the book. – pgoetz Nov 14 '12 at 23:01

The answer depends on if the data is collected by probabilistic sampling or not.

If this is a case of probabilistic sampling, then there are many good books in the field of survey sampling that could help you. The best to start with would be Lohr (2010) Sampling: Design and Analysis, 2nd Edition or Särndal, Swensson, Wretman (1992) Model Assisted Survey Sampling.

If data is not collected by probabilistic sampling (the inclusion probabilities of units in sample are not known) this is another case. I have a feeling from your description this could be your case. Propensity score weighting could be the tool for you (it is used in web surveys). There are papers by D.B. Rubin and others - for example paper by Lee (2009).

share|improve this answer

if your spouse is doing his/her analysis on a sample of the collected data, then to adjust the results for oversampling you would have to know the proportions in the population of your target variable. It would help a lot if you told us what the Y variable is and how it's measured.

For a binary target, if your sample is 50% good, 50% bad, and you knew that in your population the proportion is 5% bad and 95% good

... (presuming all bads in the population are found in the sample),

then a weight variable (for the sample) would look like this:

WEIGHT (Y=1) = 1,

WEIGHT (Y=0) = p(good)/p(bad),

where p(good) and p(bad) are the proportions of Y in the population.

For a detailed description read Naeem Siddiqi "Credit Risk Scorecards"

share|improve this answer
    
PS: Reject Inference refers to credit scoring and specifically to the population that was rejected when applied for a credit. In order to build a robust model for everyone that passes through the door, this population needs to be included with the Accepts in order to generalize the model. – Ms. L Mar 24 at 10:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.