Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite
2

This seems like a trivial question yet my lack of distributed training is leading me towards potentially more confusing answers. Hence I would like to field my question here:

I have data on several hundreds of thousands of survey respondents who received health care in 2011 and demographic information on them and their practitioner. We are doing an exploratory analysis to compare whether or not the concordence between these factors leads to an overall greater or lesser degree of satisfaction.

A descriptive table which we'd like to produce are rates of satisfactory responses, tabulated across some demographic factors and indicators of whether or not they fall above the "mean". The problem we face is that raw proportions are still so large in sample size, all response profiles show a significantly different proportion of positive responses despite being numbers which are clearly congruent (a 67 vs 68% difference).

I can think of many approaches, but cannot find literature or hard evidence towards any one approach. We are not interested in prediction intervals. Standardizing positive responses to a rate with common denominators (e.g. rate of positive responses per 100 surveys per year) makes sense, but I still see that the large sample sizes will lead to unrealistically small standard errors for those rate estimates. Adjusting for multiple comparisons makes sense, but for the wrong reasons: we're interested in tagging clinically significant effect sizes, so it's hard to justify effect-size cut-offs based on multiplicity in achieving that.

How does one with a large sample size perform comparisons with sensible alpha-levels and effect size differences?

share|improve this question
One caution I would have is that satisfaction ratings cannot be treated as true numbers that always measure the same thing in the same way. Different cultures will use scales differently, both because they may interpret the meaning of the thing to be rated differently, and because they may interpret the range of numbers on a scale differently. It is entirely possible that interesting differences between demographic profiles are simply these cultural effects and not a true difference in satisfaction. – Jonathan Jun 19 '12 at 15:33
I agree, though that to me has more to do with the interpretation of results. We are essentially looking for subgroups in which there are heterogeneous rates of reporting and might conclude post-hoc that "groups X and Y demonstrated different satisfaction rates" implying that they either get different service or have different attitudes about the same service. What I'm asking is how does one make a non-data-driven comparison of rates in the presence of big data. – AdamO Jun 19 '12 at 16:23

1 Answer

up vote 3 down vote accepted

I think the answer is very simple. The large sample size is a blessing. Don't be upset with it. The standard errors are realistic. The problem you have is that you are think of a traditional null hypothesis that the difference is exactly 0 and the alternative is that it is statistically significantly different from 0. But you are not stuck with that null hypothesis! Say that only a difference greater than 5% is clinically meaningful. Then the null hypothesis is |p1-p2|<0.05 versus the alternative that it is greater than 0.05. I took 0.05 hypothetically. Pick any delta that you consider to be clinically meaningful. You are blessed with enough data to reach a meaningful conclusion!

share|improve this answer
(+1) The "difference is exactly 0" problem is used as an example far too often. In practice, the kind of hyptheses that you propose are often of much greater interest. – MånsT Jun 19 '12 at 7:52
Thanks for that comment MansT. I think the problem may relate to the way hypothesis testing is generally taught in introductory classes. Composite null hypotheses are rsarely mentioned. I think sample size determination issues should also be given greater emphasis in elementary classes. – Michael Chernick Jun 19 '12 at 11:36
By delta, do you mean some kind of non-centrality parameter, that is that we are restricting the significance region of the parameter space to fall beyond a ball around the null hypothesized region of the parameter space? Sorry to use the math lingo, I'm trying to visualize this. How does one calibrate a statistical test to maintain the correct size when you add a "meaningful difference" requirement to the test statistic and its significance? – AdamO Jun 19 '12 at 16:26
@AdamO The idea is that you have a composite null hypothesis and a composite alternative. You only reject the null hypothesis when you have evidence that |p1-p2|> ∆. When you use a pint null hypothesis the null hypothesis is p1-p2=0 verus the alternative that p1-p2 is different from 0. With the composite null any p1-p2 within a ∆ neighborhood of 0 is acceptable and you only reject when the evidence suggests that p1-p2 is outside the interval. So the null distribution is still based on the estimate of |p1-p2|but instead of being centered at 0 it is centered at ∆. – Michael Chernick Jun 19 '12 at 17:38
The critical value is then based on the sample size and the tail area of this null distribution rather than the one that would be centered at 0. – Michael Chernick Jun 19 '12 at 17:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.