2
$\begingroup$

I have a dataset of 60,000 values from a numerical simulation.

I am testing the fit of 4 theoretical PDFs (gamma, beta, Weibull, and lognormal) to this data, with the parameters of the distributions estimated from the data.

Can I use the Chi-square statistic to compare the goodness-of-fit of these distributions? Or is the Chi-square statistic not suited for this purpose, especially given the large size of my dataset?

Improve this question
$\endgroup$
12
  • 2
    $\begingroup$ Welcome to Cross Validated! What issue(s) do you see with the large sample size? Tests tends to get more accurate as the sample size increases. $\endgroup$
    – Dave
    Commented Aug 16 at 17:29
  • 4
    $\begingroup$ See stats.stackexchange.com/q/108911/164936 . If you get a small p-value because your sample is large but the differences are small, determine if these small differences matter for the purpose of your study. Why are you testing the fit of those distributions in the first place? $\endgroup$
    – J-J-J
    Commented Aug 16 at 17:58
  • 4
    $\begingroup$ Now, it's probably not the best idea to use a chi-square goodness-of-fit test, but this is not because your sample is large (but rather because it implies binning continuous data, which loses information). I think there are several questions and answers about that on this website, you can find them if you use the local search engine (I can't look it for you right now, I'm in a hurry!). $\endgroup$
    – J-J-J
    Commented Aug 16 at 18:04
  • 2
    $\begingroup$ I would recommend quantile-quantile plots for this purpose, although much depends on how often you're doing this and on how far the person in charge (perhaps you) needs/wants/demands a formal test. With a sample size that large even well-fitting distributions won't usually pass significance tests. $\endgroup$
    – Nick Cox
    Commented Aug 17 at 9:11
  • 1
    $\begingroup$ What do you mean by "testing the fit"? For a statistically trained readership, this sounds as if you want to perform a hypothesis test, which is somewhat pointless with large datasets, because the test will indicate a "significant difference" however small it may be. Or do you want to find out which model fits best? In the latter case, the chi-square statistic is one option, but it has the disadvantage of ignoring the number of fitting parameters, i.e. how well each model in principle can be fitted to arbitrary data. $\endgroup$
    – cdalitz
    Commented Aug 17 at 19:44

2 Answers 2

Reset to default
6
$\begingroup$

When you use the data to estimate the parameters of a distribution, then the usual simple goodness-of-fit tests (including the to-be-avoided chi-square, which requires binning the data and thus losing information) don't give appropriate p-values, as those tests assume that the expected values are from a pre-specified known distribution. In any event, as you suspect, with that size of data set you might well get "statistically significant" differences from any parametric distribution.

If you just want to determine which of a set of distribution families works best to describe your data, then you could decide what criterion of "best" works best for your application (e.g., mean-squared deviation, absolute deviation, Akaike Information Criterion, ...) and choose among them accordingly. You might want to repeat the process on multiple bootstrapped samples of the data to increase your confidence (in the colloquial sense) that your results will extend well to new, similar data.

This page contains a lot of useful information, and links to more. This answer in particular illustrates several tools for numerical and graphical evaluation of distribution fits, via tests designed for continuous data, including code that uses simulations and resampling to handle the situation in which you have estimated parameters of a distribution from the data.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for the detailed response! This really helps clarify some of the doubts I had. Since my goal is to determine which one fits best, I'll try using one of the criteria you suggested, such as the Akaike Information Criterion. $\endgroup$
    – user111009
    Commented Aug 18 at 19:40
5
$\begingroup$

Can you use $\chi^2$ test? Yes, it may work (but see later in the answer). But for that you will need to bin what are continuous theoretical PDF's or CDF's, as well as the data in your sample, and that is not a good idea; you will lose a lot of information/resolution.
Now, there are tests specifically designed to test whether a sample may belong to a theoretical distribution, such as the Anderson-Darling test (A-D), or the Kolmogorov-Smirnov test (K-S). They would be much better for your purpose.
Does the large size of your dataset create a problem? Maybe? Let's start with real-world data; all real world data is bound. This is because of the quantity you are measuring (human heighths do not go to $+\infty$, or go negative; nor do blood concentration of an analyte; nor does the weighths of various objects, etc.), but also because of limitations of your measuring instrument (it has a limited range). Your real world data is also "discrete"; you record results with a finite, limited number of significant digits. In these conditions, when you have a very large dataset, and you test against a theoretical PDF, there will not be extreme values, there will be ties, all things which create a departure from the theoretical PDF, and will give you a significant result, but for the wrong reasons. Now if you test against discrete PMF's, these are bound, and ties are expected, so the problem with large datasets do not occur. But in your case, your 4 PDF's to test against are all continuous.
Note that, among these 4 PDF's 3 have (0, $+\infty)$ as their support, and 1 (beta) is defined over $(0,1)$. So that alone should eliminate some of your choices.
In your specific case, you say the data comes from a simulation; so that all depends on how "good" your simulation is (how closely it simulates the real world). Will the generated data be bound? Then a scaled $Beta$ may be your only choice (the other 3 may be rejected because they do not have extreme enough values). Does your data have ties? Then the 60,000 size will be a problem; you can look at how many ties, and if that is small (relative to 60,000), this may not be too bad. Otherwise...
In the end, your best bet may be to eyeball the shape; a histogram with 60,000 data points will give you a pretty good idea of the shape of the data.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your explanation. It helped me a lot in understanding my problem and the issues that I need to consider. $\endgroup$
    – user111009
    Commented Aug 18 at 20:03
  • $\begingroup$ A few comments: 1) I did try using the A-D and K-S tests previously, but I think that due to my sample size, they were always rejecting the null hypothesis, even when the fit looked visually good. 2) My data from the simulations is defined between (0, +∞), so I’m using the Beta distribution with two additional parameters, 'loc' and 'scale' from scipy.stats in Python, to make it comparable with the other distributions. 3) I have checked, and I don’t have too many ties. In one dataset of 60,000, I have around 300. $\endgroup$
    – user111009
    Commented Aug 18 at 20:04
  • 1
    $\begingroup$ 300 ties will be too much; which is most likely why A-D and K-S always rejected the null, as you said, "even when visually, the fit was good". So you can just rely on the visual fit, or ... a wild thought. Out of your 60,000 simulated datapoints, randomly extract say 60 (w/o replacement), and run A-D. Repeat for several subsamples (smaller size, 1 or 2 ties will not matter much). Repeat several 100's of times; I bet the ones which matched visually will fail to reject a large majority of the time. $\endgroup$
    – jginestet
    Commented Aug 19 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.