0
$\begingroup$

My confusion stems from this image cross validation example

found on the Scikit-learn docs here. As far as I understand from this picture is that the entire dataset is split into two train and test sets and cross-validation happens on the training part only. But how do we perform the final evaluation as shown in the picture by the orange box? The docs only show this example

from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1, random_state=42)
scores = cross_val_score(clf, X, y, cv=5)
scores

which would then just output the array of scores of each folds like so array([0.96..., 1. ..., 0.96..., 0.96..., 1. ]). How do I do the Final evaluation of the originally held-out data kept for final validation as shown by the orange box in the picture? I'm not getting any clear understanding of this from the docs

$\endgroup$
2
  • $\begingroup$ Welcome to Cross Validated! Are you asking about the statistics or the implementation in sklearn in particular? $\endgroup$
    – Dave
    Jan 20 at 16:59
  • $\begingroup$ Hi..I'm asking about the implementation of cross-validation in sklearn in particular. I'm confused how to perform the final evaluation after I cross-validate the data. Do I fit and predict again on the train test split or is there any other way to do so pls? $\endgroup$
    – stealth225
    Jan 20 at 17:42

1 Answer 1

0
$\begingroup$

The documentation refer to describes how cross validation can help you make better use of your data by not dividing it into a training, test, and validation set. Instead, you can use your training data to train it with cross validation (to avoid overfitting) and you use the test data (that you didn't touch during training) for model validation.

However, by partitioning the available data into three sets, we drastically reduce the number of samples which can be used for learning the model, and the results can depend on a particular random choice for the pair of (train, validation) sets.

A solution to this problem is a procedure called cross-validation (CV for short). A test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV.

Quoted from here, just above the image you posted.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.