Balance checking in randomized controlled trials with large sample I have a survey experiment in which the treated group receives a certain prompt that the control group does not.
I also have a bunch of demographic variables that I want to check for balance between the control and the treated group. A common technique that I've seen is a t-test comparing the two groups. However, if my sample size is large, doesn't this make the t-test more statistically significant regardless of the underlying balance between my two groups?
If that's true, is there a better balance test than the t-test?
 A: The people in your study were randomly assigned to the groups. Therefore, unless there was a failure of randomization, we know by definition that the demographics are identical in the population.  
It makes no sense to test if the demographics differ.  The only two possibilities are that you get a non-significant result / make a correct decision, or that you get a significant result and thus make a type I error.  
If there is reason to believe that the demographics are relevant to the response, you can still control for them in a multiple regression model.  This would give you more power to detect differences due to your treatment.  
If you want a reference, you might read:  


*

*Senn, S. (1994). Testing for baseline balance in randomized clinical trials. Statistics in Medicine, 13, pp. 1715-1726.  

A: You cannot test balance with a t-test, this is known as the ballance test fallancy - it is descibed in (Ho et. al, 2007) (http://gking.harvard.edu/files/abs/matchp-abs.shtml) 
The main points is that: balance is a feature of the sample at hand, not some population - therefore p-values < 0.05 does not imply anything.
Further, there is no point below which imbalance can be ignored, and even small differences between groups can translate into large bias. Therefore, you should always try to model differences in the (regression) model, even if the groups had almost perfect balance.
You question of sample size is speciacally adressed in the paper. You cannot use a statistic (t-test in this case) as an objective to minimize! Given a small sample a large difference could be insignificant, and given a large sample a small difference could be significant. You are therefore better off by measuring balance directly, by looking at the distribtuion of the variables (QQ-plots).
A: Statistical significance is not the only way of interpreting a coefficient. While the large sample size makes it more likely that you will find a statistically significant difference between control and treatment groups you still need to look at the magnitude of the coefficient and determine if it is large relative to the problem at hand. Similarly, if you test for balance across many variables you will likely find some that are significantly different just by chance.
A: Repmat is correct.
If you don't have demographic balance, your t-test is not testing just the treatment effect, it's testing the combined effects of (treatment effect + demo differences).  "demo differences" = bias.
Sample size does nothing to alleviate this if the imbalance persists.
If you are stuck with imbalanced data with no principled way to legitimately clean / subset your data to acheive a balanced, representative sample, then model building is necessary to control for the imbalance.
