When using a model that assumes a specific distribution of data, I get confused about how seriously I need to check for the assumption. For example, if we use some statistical test (e.g., based on p-value) for the validation, I think we should almost always reject null if sample size is large. For example, no matter how well the size of some organisms may follow a normal distribution, it is not a normal distribution for a variety of reason (e.g., it cannot be a negative value; measured values are never truly continuous because of the precision of measurement tools, etc.). I want to know what the correct attitude is for the validation of parametric assumptions.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This is a problem with standard hypothesis testing: Given large enough samples, all tests become significant.
One (non-rigorous and informal) interpretation of $p$-values is that they tell you how reproducible your experiment is. If you get a very small $p$-value, then you have good reason to hope that another scientist running the same experiment will measure an effect with the same sign as you. On the other hand, if your $p$-value is not that small, you shouldn't count on your experiment being reproducible.
Going back to the large samples problem: with enough data all experiments become reproducible, so the corresponding $p$-values become small. But that doesn't automatically mean that the result of the experiment is interesting. You still need to look at effect sizes to decide whether you care about your results. In some sense, having a small $p$-value only gives you ``moral permission'' to look and interpret the measured effect, but it doesn't necessarily imply that you've found anything interesting.
p.s. If you want to read more about this kind of problems, here's an excellent paper by Diaconis and Efron on a similar problem with tests for independence in contingency tables.
