This seems like a trivial question yet my lack of distributed training is leading me towards potentially more confusing answers. Hence I would like to field my question here:
I have data on several hundreds of thousands of survey respondents who received health care in 2011 and demographic information on them and their practitioner. We are doing an exploratory analysis to compare whether or not the concordence between these factors leads to an overall greater or lesser degree of satisfaction.
A descriptive table which we'd like to produce are rates of satisfactory responses, tabulated across some demographic factors and indicators of whether or not they fall above the "mean". The problem we face is that raw proportions are still so large in sample size, all response profiles show a significantly different proportion of positive responses despite being numbers which are clearly congruent (a 67 vs 68% difference).
I can think of many approaches, but cannot find literature or hard evidence towards any one approach. We are not interested in prediction intervals. Standardizing positive responses to a rate with common denominators (e.g. rate of positive responses per 100 surveys per year) makes sense, but I still see that the large sample sizes will lead to unrealistically small standard errors for those rate estimates. Adjusting for multiple comparisons makes sense, but for the wrong reasons: we're interested in tagging clinically significant effect sizes, so it's hard to justify effect-size cut-offs based on multiplicity in achieving that.
How does one with a large sample size perform comparisons with sensible alpha-levels and effect size differences?