5
$\begingroup$

Background

Conventional approaches to fitting a priori models to observed data seek to find those model parameters that maximize the likelihood of the data. For more complicated models, this typically necessitates an iterative search across a reasonable parameter space, computing the likelihood of the data given each candidate parameter set and selecting from amongst the evaluated candidates the set that makes the data most likely. When comparing different families of models with regards to their ability to account for the observed data, a gold-standard for the metric of comparison is cross-validated prediction performance. That is, for each model family of interest and each sub-set of the observed data, maximum likelihood estimates of that family's parameters are obtained given the data omitting that subset, and the likelihood of the omitted subset given these parameters is evaluated. Aggregating likelihood estimates across subsets yields a measure of fit that permits comparison of model families that is uncontaminated by potential differences between families in their ability to fit noise over-and-above their ability to fit phenomena of interest in the data.

Foreground

It strikes me that this hierarchically iterated approach, searching for ML parameter estimates repeatedly for each cross-validation subset, may be computationally simplified to a one-stage approach. Specifically, I wonder if a non-self-terminating genetic algorithm like that employed by DEoptim might be supplied with an objective function that randomly samples (with replacement) the observed data before evaluating the adequacy of a given candidate parameter set of a given model. (Thus, the "search" is now for the candidate parameter set that maximizes the likelihood of data likely to be observed in the future.) After a large number of generations, the latest surviving generation of parameter values (or some aggregation of the N most recent generations) is chosen as the final estimates and the adequacy of the model (for comparison to other model families) is evaluated by computing the likelihood of the observed data (no resampling) given this model.

Questions

  1. Is this idea new?
  2. Is this idea blatantly not worthwhile? (If so, why?)
$\endgroup$
  • $\begingroup$ Note that use of resampling with replacement as an approximation to future data assumes that the observed data is representative of that of the data generating population. However, I believe this assumption is held rather broadly across statistical tools. $\endgroup$ – Mike Lawrence Jun 14 '11 at 16:05
2
$\begingroup$

The idea looks related to using the bootstrap, see Section 7.11 in ESL, as an alternative to cross-validation. The bootstrap also resamples subsets for training and uses the original data for estimation of the generalization error (evaluation of the model). The difference, as far as I can see, is that when using the bootstrap we train the model on the subsampled data sets, whereas the suggestion here is "merely" to evaluate a given parameter on a subsampled data set. The training happens inside the magic genetic algorithm (that I don't know anything about) by modifications of the parameters before the next evaluation.

For the bootstrap various issues arise from the fact that subsamples share observations with the date set used for evaluationO, and the average number of distinct observations in a subsample is somewhat smaller than the size of the data set. This leads to strange corrections like the .632-estimator, see (7.56) in ESL, and I don't think that the bootstrap has any edge over cross-validation.

I have no reason to rule out the idea presented, but I would be skeptical about the final evaluation on the full data set. As I understand the suggested idea, the algorithm will produce a sequence of parameter estimates that get better and better when evaluated on subsampled data. Because you constantly subsample, I don't see that the algorithm can be convergent, but it may stabilize after a number of iterations. I would imagine that the parameters would then be "around" the MLEs for the subsampled data (but I may be wrong here), and that there would be issues similar to those for the bootstrap with evaluating parameters/models on the full data set.

By the way, if I remember correctly, the fitting of e.g. generalized additive models by the R package mgcv uses an algorithm where everything is optimized in one go.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.