3
$\begingroup$

As for cross evaluation (CV), I have two questions to ask:

1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model?

2) In k-fold CV, what's the final model to be retained? I understand that in each fold, the model is created and them discarded immediately?

I am new to ML and get confused about the purpose of CV.

$\endgroup$
2
$\begingroup$

1) CV has nothing to do with parameter selection, but only model evaluation?

Yes, CV is about model evaluation or verification (which is a sub-task of validation) or performance estimation or performance measurement.

The link to parameter selection is that some strategies for parameter selection rely on performance estimates or measurements, and those can be obtained via cross validation.
So the relation is a rather weak (parameter selection can use other criteria instead of estimated generalization error, and even that can be estimated by other thechniques than CV) one-way type of relation.

Specifically, which model?

Any model. One advantage of cross validation is that it works completely independent of the model as it treats the model as a black box that spits out predictions when applied to input data

2) In k-fold CV, what's the final model to be retained?

Typically, k-fold CV (or any other restampling validation technique) is used to get an approximation to the generalization performance (performance of predictions for unknown cases) of the model trained on all available cases. So none of the $k$ models is retained, but the results of the k-fold CV procedure are used as approximation/estimate for the performance of yet another model (the one obtained by using the same training procedure together with all cases).

I understand that in each fold, the model is created and them discarded immediately?

Typically yes, but sometimes these so-called surrogate models are retained for further inspection (e.g. wrt. stability).
They can also be used for model aggregation, but if you do this you'll have to keep in mind that the original CV performance estimate does not estimate predictive performance of the aggregated model.

$\endgroup$
1
$\begingroup$

Perhaps I can help by explaining the reason cross-validation is used.

Motivation: the situation for each model

If $\tilde{y}$ is future/unobserved data and $y$ is data you have, a scoring rule/function $S(p,\tilde{y})$ is a function that takes

  1. the distribution you're using to forecast $p$ (estimated from $y$) , and
  2. a realized future/unobserved value $\tilde{y}$

and then gives you a real-valued number/score/utility. Higher is better, although this convention isn't always followed in the literature. This score is random because it's a transformation of a random future data point, so people are usually interested in its expected value. The expected value tells you the predictive capabilities of your model. We could write this as $$ E_f[S(p,\tilde{y})] $$ where $f$ is the unknown true distribution of $\tilde{y}$. For example, a popular scoring rule is $S(p,\tilde{y}) = \log p(\tilde{y})$. You take the future data point, and plug it into your log-likelihood.

Approximation

The trouble though is that the expected score is hard to get at because

  1. you don't know the true distribution of $\tilde{y}$, and
  2. you don't want to wait around to get samples of this future data (if you did, you could the Monte-Carlo technique and take the average score to get something really close).

Also, you can't just plug in $y$ for $\tilde{y}$ because this will overstate your models' predictive capability. This is why all of this data splitting comes into play.

So here's the point: cross-validation is a way to estimate this expected score. You repeatedly partition the data set into different training-set-test-set pairs (aka folds). For each training set, you estimate the model, predict, and then obtain the score by plugging the test data into the probabilistic prediction. This gives you one score for each fold. Then you take the average of these (or the sum, depending on what reference you're using).

A few things about CV:

  1. The partitions are nonrandom, test sets are disjoint
  2. for each split/estimation/prediction, we never use a data point twice
  3. for each split/estimation/prediction, we lose parameter estimation accuracy because each training set is smaller than the full set
  4. however, we get to average over many prediction scores, which reduces variance
  5. the estimator is still biased, so people may or may not try to correct for this
  6. it can be computationally brutal to calculate for some models
  7. the logo of this site illustrates this splitting procedure!

Many models

And yes, you are right, after this whole thing is done for multiple models, you can see which model has the the best (estimated) predictive capability. You can pick that model, and then estimate it using the full data set. Before you implement this in practice, you should know that blindly doing CV for a large collection of models will rarely help you pick the correct one, but that's a question for another thread.

$\endgroup$
  • $\begingroup$ if you just choose "the best (estimated) predictive capability" as the final model, what's the purpose of "you take the average of these"? Taking the average score of multiple models is just a calculation, but it doesn't really adjust the weights of models based on the average. It's still not clear to me why it's averaged. $\endgroup$ – ling Jan 20 at 5:08
  • $\begingroup$ @ling: The average is taken over $k$ so-called surrogate models which are assumed to be equvialent (and for [hyper]parameter selection, those surrogate models have the same [hyper]parameters) and are tested with disjoint sets of test cases. Maybe a better way to express what is done is that under the assumption that those models are equivalent, we can pool their predictions. The multiple models Taylor writes about in the last paragraph are in contrast models (bunches of k surrogate models) that differ in their [hyper]parameters. Taylor, maybe you could clarify your answer? $\endgroup$ – cbeleites Jan 20 at 15:12
  • $\begingroup$ @cbeleites so in each fold, the hyperparameters are fixed, and the best model is the 'fold' which produces the best average score and best hyper-parameters. $\endgroup$ – ling Jan 20 at 18:25
  • $\begingroup$ @ling each model gets fit on many folds. $\endgroup$ – Taylor Jan 20 at 19:06
  • $\begingroup$ @ling: no. as Taylor says, for each hyperparameter combination, you do a full cross validation of k models, and pool (e.g. average) their results. You then look for the hyperparameters that gave the optimal cross valdiation result (= pooled over its k surrogate models). Maybe you could study cross validation without hyperparameter optimization first, and once you understand that, you can study how it is used during hyperparameter optimization. $\endgroup$ – cbeleites Jan 22 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.