How to select the best fit without over-fitting data? Modelling a bimodal distribution with N normal functions, etc

Question

I have an obviously bimodal distribution of values, which I seek to fit. The data can be fit well with either 2 normal functions (bimodal) or with 3 normal functions. Additionally, there is a plausible physical reason for fitting the data with 3.

The more parameters that are introduced, the more perfect the fit will be, as with enough constants, one can "fit an elephant".

Here is the distribution, fit with the sum of 3 normal (Gaussian) curves:

These are the data for each fit. I'm not sure what test I should be applying here to determine the fit. The data consists of 91 points.

1 Normal Function:

• RSS: 1.06231
• X^2: 3.1674
• F.Test: 0.3092

2 Normal Functions:

• RSS: 0.010939
• X^2: 0.053896
• F.Test: 0.97101

3 Normal Functions:

• RSS: 0.00536
• X^2: 0.02794
• F.Test: 0.99249

What is the correct statistical test that can be applied to determine which of these 3 fits is best? Obviously, the 1 normal function fit is inadequate. So how can I discriminate between 2 and 3?

To add, I'm mostly doing this with Excel and a little Python; I don't yet have familiarity with R or other statistical languages.

Answer 1

Here are two ways you could approach the problem of selecting your distribution:

1. For model comparison use a measure that penalizes the model depending on the number of parameters. Information criteria do this. Use an information criterion to choose which model to retain, choose the model with the lowest information criterion (for example AIC). The rule of thumb for comparing if a difference in AIC's is significant is if the difference in the AIC is greater than 2 (this is not a formal hypothesis test, see Testing the difference in AIC of two non-nested models).

   The AIC = $2k - 2ln(L)$, where $k$ is the number of estimated parameters and $L$ is the maximum likelihood, $L = \max\limits_{\theta} L(\theta |x)$ and $L(\theta |x) = Pr(x|\theta)$ is the likelihood function and $\Pr(x|\theta)$ is the probability of the observed data $x$ conditional on the distribution parameter $\theta$.

2. If you want a formal hypothesis test you could proceed in at least two ways. The arguably easier one is to fit your distributions using part of your sample and than test if the residuals distributions are significantly different using a Chi-squared or Kolgomorov-Smirnov test on the rest of the data. This way you're not using the same data to fit and test your model as AndrewM mentioned in the comments.

   You could also do a likelihood ratio test with an adjustment to the null distribution. A version of this is described in Lo Y. et al. (2013) "Testing the number of components in normal mixture." Biometrika but I do not have access to the article so I cannot provide you with more details as to how exactly to do this.

   Either way, if the test is not significant retain the distribution with the lower number of parameters, if it is significant choose the one with the higher number of parameters.