Testing samples against a distribution

Question

Suppose I have some data sitting in front of me, which is supposed to be sampled from a specific distribution. How would I go about testing whether it is from that distribution?

I've seen several tests which appear to do this - Pearson's chi-squared test, the one-sample Kolmogorov-Smirnov test, and the Anderson-Darling test, to name a few. Which one do I want here? (And if the answer is "it depends", then what does it depend on?)

Answer 1

There are many that can be used and the depends is that each look at different attributes of the distributions, so which is best depends on how you think your data is most likely to differ from the null distribution. One option would be to use several of them (to capture the different ways they might differ), but be sure to adjust for multiple comparisons. Also note that all the tests are rule out tests, if they are not significant that does not mean that the data comes from that distribution, just that you cannot rule it out. Also note that these tests often have low power to rule out many distributions at small sample sizes and with large sample sizes can find differences statistically significant that would not be considered practically significant.

One test that gives you more options to look at meaningful differences is the visual test described in:

Buja, A., Cook, D. Hofmann, H., Lawrence, M. Lee, E.-K., Swayne,
 D.F and Wickham, H. (2009) Statistical Inference for exploratory
 data analysis and model diagnostics Phil. Trans. R. Soc. A 2009
 367, 4361-4383 doi: 10.1098/rsta.2009.0120

Which consists of making several plots of data simulated from the null distribution along with the plot of the actual data, then see if someone not familiar with the data can tell which is the real data. This is implemented in the vis.test function in the TeachingDemos package for R or can be easily implemented using other tools.