Conventional approaches to fitting a priori models to observed data seek to find those model parameters that maximize the likelihood of the data. For more complicated models, this typically necessitates an iterative search across a reasonable parameter space, computing the likelihood of the data given each candidate parameter set and selecting from amongst the evaluated candidates the set that makes the data most likely. When comparing different families of models with regards to their ability to account for the observed data, a gold-standard for the metric of comparison is cross-validated prediction performance. That is, for each model family of interest and each sub-set of the observed data, maximum likelihood estimates of that family's parameters are obtained given the data omitting that subset, and the likelihood of the omitted subset given these parameters is evaluated. Aggregating likelihood estimates across subsets yields a measure of fit that permits comparison of model families that is uncontaminated by potential differences between families in their ability to fit noise over-and-above their ability to fit phenomena of interest in the data.
It strikes me that this hierarchically iterated approach, searching for ML parameter estimates repeatedly for each cross-validation subset, may be computationally simplified to a one-stage approach. Specifically, I wonder if a non-self-terminating genetic algorithm like that employed by DEoptim might be supplied with an objective function that randomly samples (with replacement) the observed data before evaluating the adequacy of a given candidate parameter set of a given model. (Thus, the "search" is now for the candidate parameter set that maximizes the likelihood of data likely to be observed in the future.) After a large number of generations, the latest surviving generation of parameter values (or some aggregation of the N most recent generations) is chosen as the final estimates and the adequacy of the model (for comparison to other model families) is evaluated by computing the likelihood of the observed data (no resampling) given this model.
- Is this idea new?
- Is this idea blatantly not worthwhile? (If so, why?)