Model selection for nonlinear regression of a Gaussian CDF mixture distribution

Question

I have a number of distributions which I want to fit to a CDF that is comprised of one or more Gaussian CDFs. I was able to use weighted least squares regression to find the best fit parameters for each regression depending on the number of modes I give the model (e.g. each Gaussian CDF has three parameters, mu, sigma, and a scaling factor, that are summed to create my total function). In addition, it is reasonable that the CDF may not actually approach 1. If the fit places the CDF between approximately 0.8 and 1.2, it would be physically justifiable. I've included example plots of my 1, 2, and 3 mode fits for a distribution (Ignore scale on left of plots).

I am trying to figure out if there is a good way to determine if one model is better than another without overfitting. My thought was to try using AIC, but I'm not sure the methods I'm using are the correct way of thinking about this problem or not.

Currently, I am minimizing the weighted least squares by minimizing: $$ \sum_i [w_i * (f(x_i) - y_i)^2] $$ Where $w_i$ is the weight for each datapoint. I'm not getting confused though, as what I'm finding seems to indicate that log-likelihood would be something like: $$ log L = \sum_i \frac{w_i * (f(x_i) - y_i)^2}{2\sigma^2} $$ Is this actually the log-likelihood? Why is it not being maximized? And additionally, I'm trying to understand how this fits in to AIC or AICc in order to penalize the additional parameters with the multiple mode fits. Does anyone have suggested resources for better understanding model selection for nonlinear regression? Most of the information I have found has been on Maximum Likelihood Estimation for probability distributions rather than this sort of sigmoid curve fitting.

Answer 1

AIC (or AICc) is commonly used for selection of the preferred model. This reference (http://www.mun.ca/biology/quant/ModelSelectionMultimodelInference.pdf) has some good background information.

The AIC = logL + 2*k (where k=number of parameters). So, the AIC maximizes the log-likelihood but then penalizes for each additional dimension (aka parameter) that is available to the problem. A simple way to think about it (not quite correct, but it shows the message) is that with each parameter, you add a dimension for fitting, and in that dimension you can go two directions relative to your starting point (up or down).

The AIC collapses to a chi-squared test with k degrees of freedom when you add one parameter to a nested model with a p-value of 0.15.

Sometimes variants on the AIC are used that include a different per-parameter penalty than 2 (my preference is 6.63 which equates to the chi-squared test with one degree of freedom and p=0.01).