Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 19 down vote favorite

What if you take a random sample and you can see it is clearly not representative, as in a recent question. For example, what if the population distribution is supposed to be symmetric around 0 and the sample you draw randomly has unbalanced positive and negative observations, and the unbalance is statistically significant, where does that leave you? What reasonable statements can you make about the population based on a biased sample? What is a reasonable course of action in such a situation? Does it matter when in our research we notice this imbalance?

share|improve this question
2  
Michael, this problem might be expected to occur one time in 20, if we use statistical significance as our metric. Most often we do not know when we have randomly chosen a non-representative sample because we do not know enough about the population. But when we do know something about the population, and we notice such an anomaly, what do we do? – Joel W. Jul 16 '12 at 13:26
2  
Yes, the most correct practice is to obtain a large enough random sample, like @MichaelChernick wrote. However, one of my professors tell me that he verified by Monte Carlo simulation that, when a researcher has to increase the sample size, it is not so correct to simply add statistical unities to the sample, but one has to repeat the sampling. Otherwise, the statistics may be biased (once again!). – this.is.not.a.nick Jul 16 '12 at 13:27
4  
@Michael, I do not understand why your statement is true. A p-value less than .05 will occur under the null hypothesis 5% of the time regardless of sample size. So how can it be possible that larger sample sizes will solve this problem? It seems to me your recommendation implicitly invites readers to confuse the size and power of hypothesis tests. – whuber♦ Jul 16 '12 at 13:34
2  
@Michael, what do you mean that we should collect more data at random? Are we to hope that we randomly draw a sample biased in the other direction? In any case, what number of additional cases should we draw? Do you suggest we set a number at the onset or use a stopping rule? If a stopping rule, what might the rule look like? Finally, even if the resulting larger sample has no statistically significant bias, we know it is comprised of two samples, one with bias and one without. What reasonable statements can you make about the population based on such a complex sample? – Joel W. Jul 16 '12 at 13:39
2  
@Michael An alternative conclusion is that a highly significant, highly skewed sample indicates a problem with the sampling procedure. If so, the lack of symmetry will persist in a larger sample. – whuber♦ Jul 16 '12 at 13:47
show 11 more comments

3 Answers

up vote 5 down vote

The answer given by MLS (use importance sampling) is only as good as the assumptions you can make about your distributions. The main strength of the finite population sampling paradigm is that it is non-parametric, as it does not make any assumptions about the distribution of the data to make (valid) inferences on the finite population parameters.

An approach to correct for sample imbalances is called post-stratification. You need to break down the sample into non-overlapping classes (post-strata), and then reweight these classes according to the known population figures. If your population is known to have a median of 0, then you can reweight the positive and negative observations so that their weighted proportions become 50-50: if you had an unlucky SRS with 10 negative observations and 20 positive observations, you would give the negative ones the weight of 15/10 = 1.5 and the positive ones, 15/20 = 0.75.

More subtle forms of the sample calibration do exist, in which you can calibrate your sample to satisfy more general constraints, such as having a mean of a continuous variable to be equal to the specific value. The symmetry constraint is pretty difficult to work with, although that might be doable, too. May be Jean Opsomer has something on this: he has been doing a lot of kernel estimation work for survey data.

share|improve this answer
How does post-stratification compare, logically or statistically, to simply discarding the unbalanced sample and drawing another sample? (Sometimes drawing the sample is the labor intensive part of the research, but sometimes it is what is done after you have drawn the sample that is labor intensive and drawing the sample involves relatively minor effort, as in much experimental research.) – Joel W. Jul 16 '12 at 18:37
1  
I have never been in a situation where discarding the data is the best answer, and I have never seen it discussed in any of the survey statistics books. In most of survey statistics, getting the data is at least five times more expensive than any of the following data processing and analysis (except probably for some cheap web surveys where the data collection is nearly free). If you are in an experimental world, then you should not tag your post "sampling", and rather use "experiment design" instead. – StasK Jul 17 '12 at 17:06
up vote 1 down vote

This is a partial answer that assumes we know both the distribution $q$ from which was sampled, and the true (or desired) distribution $p$. Additionally, I assume that these distributions are different. If the samples were actually obtained through $p$, but they look wrong: the samples are still unbiased and any adaptation (such as removing outliers) will likely add bias.

I assume you want to find some statistic $s_p = E \{ f(X) | X \sim p \}$. For instance, $s(p)$ might be the mean of the distribution, in which case $f$ is the identity function. If you had samples $\{ x_1, \ldots, x_n \}$ obtained through $p$, you could simply use $$ s_p \approx \frac{1}{n} \sum_{i=1}^n f(x_i) \enspace. $$ However, suppose you only have samples that were obtained (from the same domain) with a sampling distribution $x_i \sim q$. Then, we can still get an unbiased estimate of $s_p$ by weighting each of the samples according to the relative probability of it occuring under each distribution: $$ s_p \approx \frac{1}{n} \sum_{i=1}^n \frac{p(x_i)}{q(x_i)} f(x_i) \enspace. $$ The reason this works is that $$ E \left\{ \frac{p(X)}{q(X)} f(X) \middle| X \sim q \right\} = \int p(X) f(X) dx \enspace, $$ as desired. This is called importance sampling.

share|improve this answer
You say the sample is not biased and any attempt to fix the sample will add bias. I suggest that the process by which the sample was collected is without bias but, in fact, the sample is biased, perhaps seriously biased. Are there ways to try to fix the known large bias that might be expected to introduce relatively little additional bias? – Joel W. Jul 16 '12 at 13:54
1  
To disambiguate the terminology a bit: I think of bias as a property of the expectation of a random variable. In other words, if the process that collects the data is unbiased, then so is the sample. However, the sample may still be atypical and lead to unwanted conclusions. Any general way to fix this induces bias, since you are adapting the (unbiased) sampling procedure. Probably the less biased approach is to collect and use new samples. A slightly more biased approach would add these new samples to the old ones, but the result might be less variable since you have more samples in total. – MLS Jul 16 '12 at 14:10
2  
@Joel W. What do you mean when you say the sample is biased? Is it the estimate of the mean based on the sample that is biased? Any sample estimate is going to differ from the true mean and some can be far off. When sampling at random this is due to variance not bias. It is not right to say a sample is biased because the distribution of the sample is known to look a lot different from the distribution for the population. In small samples many can look unrepresentative for one reason or another but random sampling is not biased sampling. – Michael Chernick Jul 16 '12 at 14:19
1  
@Michael, I agree that we must recognize and live with random variance when we have to. I am asking what we might reasonably do when we detect unintended variance. What if our random sample turns out to include relatively too many young people, or too many blue collar workers, etc., when those categories are relevant to our research? Going even further, should we check our samples to see if they are unbalanced in such ways? And does it matter if we notice this before doing further research with the sample or after we have invested resources in conducting research with the sample? – Joel W. Jul 16 '12 at 15:20
1  
Covariate imbalance is very important. If it exists in a sample a regression model can be used to adjust for it. Vance Berger has written a book on this topic which I have probably cited previously on this website. Here is an amazon link to a description of the book. amazon.com/Selection-Covariate-Imbalances-Randomized-Statistics/… – Michael Chernick Jul 16 '12 at 15:30
show 5 more comments
up vote 1 down vote

I'm the Junior Member here, but I'd say that discarding and starting over is always the best answer, if you know that your sample is significantly unrepresentative, and if you have an idea of how the unrepresentative sampling arose in the first place and how to avoid it if possible the second time around.

What good will it do to sample a second time if you'll probably end up in the same boat?

If doing the data gathering again doesn't make sense or is prohibitively costly, you have to work with what you have, attempting to compensate for the unrepresentativeness via stratification, imputation, fancier modeling, or whatever. You need to clearly note that you compensated in this way, why you think it's necessary, and why you think it worked. Then work the uncertainty that arose from your compensation all the way through your analysis. (It will make your conclusions less certain, right?)

If you can't do that, you need to drop the project entirely.

share|improve this answer
What if you do not know why the sample is unrepresentative, are you still justified in discarding it and drawing a new, random sample? If not, why not? Also, let's say you do discard the first sample and draw a second one, are the inferential statistics that you might calculate based on the second sample in any way inappropriate due to the discarded first sample? For example, if you subscribe to discarding unrepresentative samples, are you changing the sampling distribution that your statistical test is based on? If so, are you making it easier or harder to find statistical significance? – Joel W. Jul 16 '12 at 22:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.