0
$\begingroup$

My past assumption of cross-validation (in particular k-fold CV) was that in order to given same chance to each sample in our dataset to appear in training , we use k-fold CV. Under my assumption we define our classifier/model (select the type of classifier/model and set the classifier/model parameter) and test the performance of it on different test dataset by using k-fold, but after reading some posts, I found that we use k-fold CV in order to choose which one of our trained classifier/model is best and so we would choose it as our final classifier/model (which set of model parameter lead to better performance).

So now I have a these two questions:

First, which one of this is correct?

  1. in k-fold CV, our classifier/model is set, and we only use k repetition to obtain better (more accurate) evaluation of our classifier/model performance (for example accuracy).

  2. in k-fold CV, our classifier/model is not set (the type of classifier/model is selected (for example LDA) but the selected classifier/model's parameter are not set), and we only use k repetition to identify which classifier/model lead to better performance so we select that one as our final classifier/model. The performance measurement is not the main focus here.

Second, if the second definition is correct one, how can we explain the final averaging in k-fold cross-validation: how can a person average different classifier/model (a same classifier/model with different parameter value)?

$\endgroup$
2
$\begingroup$

Cross-Validation is really only a way to get a realistic sense of a process performance. What you are doing during k-folds or LOO is test the process by which you get to your model on several datasets so as to get a realistic performance measure.

For exemple, let's take a simple regression model $\mathcal{M}$ and a dataset $\mathcal{D}$. If you train $\mathcal{M}$ on $\mathcal{D}$ and then try to test the performance of $\mathcal{M}$ for prediction then you will get overly optimistic results as $\mathcal{M}$ was trained on $\mathcal{D}$ so you basicaly ask $\mathcal{M}$ to predict what it knows.

That is a problem because $\mathcal{M}$ may be overfitting $\mathcal{D}$, meaning that it perfectly describes $\mathcal{D}$ and if by any chance $\mathcal{D}$ is not representative of all cases you would encounter then the predictions of $\mathcal{M}$ may be totally off later without you even knowing.

To get a realistic performance measure you have to separate the data on which you learn and the data on which you test. Hence $\mathcal{D}$ gets separated in two part $\mathcal{D}_L$ and $\mathcal{D}_T$, $\mathcal{D}_L$ for learning and $\mathcal{D}_T$ for testing.

But now that you have done that your choice for $\mathcal{D}_L$ maybe a 'bad one', meaning that particular subset of $\mathcal{D}$ may be specifically bad while any other would have yield greater results.

That's why we have k-folds you divide $\mathcal{D}$ in multiple parts ($\mathcal{D}_k$) and each of these $\mathcal{D}_k$ will be used as testing grounds for a model fitted on the other $\mathcal{D}_{i\neq k}$. That way you can have a more realistic measure of your model's real performance for prediction.

The most extreme version of k-folds is Leave-One-Out (or LOO), where $k = nrow(\mathcal{D})$ and you fit your model on all your observations but one and predict for that observation alone. It is handy when $\mathcal{D}$ is small and you don't want to fit your model on a too small learning dataset.

As a rule of thumb the greater $k$, the more folds, the better your measure will be. But that is not always the most clever thing to do as when $\mathcal{D}$ is large the computing time will be huge while the true gain in 'accuracy' will remain modest.

So yeah, cross-validation is not a way to get the better model bu a way to assess the validity of a process. I'm using process because it can be used for about anything else as well and also because if you have data transformation, variable selection, etc... going in your model fit then you would have to include those parts in the cross-validation.

Hope it helps and I was clear,

$\endgroup$
1
$\begingroup$

I actually do something like k-k-fold validation! I have noticed (especially with smaller datasets) that the results on a holdout/validation set after CV can vary quite a lot (for example, a model might show AUC ranging from 0.55 to 0.7 on 5 different training iterations even in a 10-fold CV). In this case one might decide to take the "best" model as the one that generated 0.7 AUC on a hold out set (selected randomly from the original full data set), but in the real world, I would think actual performance is going to be more like the average of all these models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.