2
$\begingroup$

I have a data set 1035 x 16, implemented random forest regressor algorithm after all the possible feature processing techniques, test/ train split is 20/80, for cross validation I used 5 K-Folds and following are the results:

Test Score : 72.82762577833321
Training Score: 97.38484784016136
Validation score : [0.63171505, 0.64145713, 0.67557705, 0.66368883, 0.6085836 ]
Validation score avg : 64.42043307160208

I know model is over-fitting but how I can interpret, the Cross Validation score is less than Test score .

when the Cross Validation score is less than Test score ? when the Cross Validation score is greater than Test score ? which is better ?

$\endgroup$
1
  • $\begingroup$ Are there any differences between how you make predictions when validating vs testing? $\endgroup$
    – Tim
    Mar 21, 2022 at 14:04

5 Answers 5

2
$\begingroup$

The same condition happened to me many times. I believe it is normal to get such results, as cross validation score is basically a "test score". It is also the average of the k-folds so it is a more stable score than just the final test score, which is a single score. So, for me it is okay to get such result. But I fear it is a little an opinion, rather than a concrete answer.

$\endgroup$
1
$\begingroup$

A high training score and low validation score signals overfitting as you’ve also mentioned. And, overfitting means your test results is subject to high variance, which naturally explains the large difference between the test and validation sets.

$\endgroup$
1
$\begingroup$

Training score is more than the validation score when the model overfits. Typically, the validation score is less than the training score, because model fits on training data, and validation data is unseen by the model.

Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data.

But sometimes validation score can be more, it just means your validation data is easier to classify. In this case, try to take a bigger validation dataset, or use multiple fold cross-validation and this case should disappear.

I hope this helps.

$\endgroup$
2
  • $\begingroup$ I am asking for test and validation score $\endgroup$
    – zack
    May 1, 2020 at 1:54
  • $\begingroup$ @zack Generally, the term “validation data” is used interchangeably with the term “test data” and refers to a sample of the dataset from training the model. You typically use cross-validation just to find the best hypermeters for the model to train(on train data). Hence, it would not make much sense to compare the cross-validation score and test scores. You assess your model performance based on train and test scores. $\endgroup$ May 1, 2020 at 2:34
0
$\begingroup$

I ran into a problem as this recently. It is possible that there is some innate order in your dataset, e.g. time series, or rows are sorted by a certain column. When you do train-test split, the dataset is shuffled by default. However, cross_validate do not shuffle dataset by default.

To avoid the problem, you can use shufflesplit when generating cross validation data splits.

cv = ShuffleSplit(n_splits=5, test_size=0.25, random_state=0)
cv_results = cross_validate(model, data, target, cv=cv)

or any other way to explicitly tell cross_validate to shuffle dataset before split, e.g. use K-Folds cross-validator and pass shuffle=True.

$\endgroup$
0
$\begingroup$

Depending on your dataset splitting, It is assumed you used 80% for the training and 20% for the test sets. You used 5-fold cross-validation with your training set (80%). So, it means the validation set is a set that was split from the training set in k-fold processing.

Your training and validation results show that your model is over-fitting (as you mentioned).

Cross-validation scores might be lower than test scores based on some conditions:

  • Small size of dataset.
  • Training data does not represent the validation data.
  • Training data represents the test set rather than the validation set.
  • The test set contains easier data to predict than the validation set.

Cross-validation scores might be higher than test scores if you can solve the condition above and some more conditions are considered:

  • Complexity of your model.
  • Try to shuffle your data when splitting it.
  • Ensure your train and test set come from the same distribution.
  • Add drop-out or regularization layer.

Even most people said a validation score higher than a test score is considered good model learning. But no one can guarantee it. A good model can result in higher testing accuracy than the validation accuracy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.