Method(s) to avoid overfitting model parameters for pre-determined model structure?

Question

I'm using maximum likelihood estimation to fit a model of a pre-determined form to some data. To test this fitting method, I decided to generate some simulated data using the precise model form and known parameters to determine if I could recover those parameters. When I do this using roughly the amount of simulated data as I will be using when fitting the actual data (~500 data points to fit 7 parameters), I find that the fit model parameters often do not generalize well to explain new simulated data sets. In other words, the model seems to be overfitting.

I'm looking for advice on how to improve my fitting procedure. I've seen a lot of resources out there about cross-validation methods, though I seem to only see this being used for model selection, not for estimating parameters of a model where the form has already been chosen. Is it also used in the case I'm describing? I would imagine you would divide your data into a number of training and test subsets, train on a bunch of different subsets, and choose the set of parameters that produce the highest likelihood for the test set. Is this a principled way of finding the best set of parameters? If not, what would be better? Thanks for your help!

Answer 1

I'm looking for advice on how to improve my fitting procedure. I've seen a lot of resources out there about cross-validation methods, though I seem to only see this being used for model selection, not for estimating parameters of a model where the form has already been chosen. Is it also used in the case I'm describing?

You are correct it is mostly used in model selection but cross validation is also used in the case you are describing (to estimate the parameters of the model). This is often known as Model Structure Selection. The model structure selection is the application of statistical methods for selecting the parameters and hyperparameters of the model. For example, given a machine learning problem, we choose to model it with polynomial curve fitting assuming some prior knowledge that the model is not linear. Even in polynomial curve fitting: the choice among ax+b, ax^2+bx+c and other higher order polynomials is an arduous task. The concept of underfitting, over-fitting, bias-variance dilemma (trade-off) are the central issues to be considered in model structure selection.

I would imagine you would divide your data into a number of training and test subsets, train on a bunch of different subsets, and choose the set of parameters that produce the highest likelihood for the test set. Is this a principled way of finding the best set of parameters? If not, what would be better? Thanks for your help!

The methodology or the process you have defined is correct. We can the cross-validation to train the model of different complexities and select the parameter that generalizes best, i.e., produces best results on the test data. Since you have 7 parameters grid search may be the way to go. Example of this approach can be you have selected a model (clustering method) but would like to determine the number of clusters in the data. Like you see here you see here, you can use cross-validation to select appropriate number of components.

Regarding overfitting: you only have 500 samples and 7 parameters. According to the rules of thumb[1,2,3], these numbers should be adequate but I feel they are still limited for model structure selection problem, more data is always better. Since you are generating simulated data, why not try with different sample size and study their effects in parameter selection.