I'd like to know if there are stricter alternatives to automated model selection than AICc / AIC / BIC.
We have approximately ten thousand curves, and for each we'd like to find the most parsimonious Gaussian mixture model. Our maximum complexity is a mixture of 5 Gaussians. The procedure so far is to fit these 5 models:
gauss1 = a1*exp(-((x-b1)/c1)^2) gauss2 = a1*exp(-((x-b1)/c1)^2) + a2*exp(-((x-b2)/c2)^2) ... gauss5 = a1*exp(-((x-b1)/c1)^2) + ... + a5*exp(-((x-b5)/c5)^2),
and then calculate AICc for each and choose the model with the lowest AICc value. We repeat this for each of the ten thousand curves.
To verify whether it's working, I generated some simulation data which looks a lot like our real data. The simulated data is generated by models gauss1, gauss2, ..., gauss5 + noise. To include noise, I added a random number to each point (between 0 and 10% of the maximum value of the simulated data, uniform distribution). This way I know what the model complexity should be.
Unfortunately it looks like AICc is choosing overly complex models. An example fit for simulated data is below. The complexity of the generating model is 2 Gaussians (gauss2) and the best fitted model, as chosen by AICc, was gauss5.
Using 50 simulated curves, here is the model complexity breakdown. The simulated curves were chosen to be biased toward a simpler model. However the complexity of the fitted models, as chosen by AICc, is biased toward the more complex. For example from these 50 curves, AICc never chose gauss1, even though 19 out of 50 were generated by gauss1. Similar results are seen for AICc, AIC, and BIC.
Choosing between the 5 models with AICc (or AIC or BIC) does not look like it's working for us. Are there other accepted ways to pick between a small number of models? I don't think I can use LASSO / ridge regression since I believe that's only for linear regression. I also don't want to build something ad hoc.