2
$\begingroup$

I know that cross validation helps reduce overfitting in the data by its property of testing and training on all the instances. But is there any chance that it also reduces underfitting? I mean, if we have fewer instances in data and we don't want our model to be more general (underfit). We can use cross validation to reduce the effect of underfitting as well. But I am not sure about it. Please let me know if that's correct what I am thinking and if cross-fold-validation reduces underfitting as well.

$\endgroup$
6
  • $\begingroup$ If you have not enough data, do you think cross validation could help you ? You will adjust you model on fewer data .. so how could it be an help against the lack of information ? $\endgroup$ Jun 29 at 8:19
  • 3
    $\begingroup$ Your assumption is wrong. Cross-validation does not reduce overfitting per se. It just tries to give a rough guess of true performance. Reducing over- or underfit always requires to change your model. $\endgroup$
    – Michael M
    Jun 29 at 8:29
  • $\begingroup$ @MrSmithGoesToWashington well, it would, by some margin helpful. You know that cross-validation works in a way that let's say I have 10 instances rather than splitting it in a way that I have 6 instances for training and 4 for testing (train-test-split). I use cross-validation that would train on the first 9 and test on the 10th one and it goes the same way until all the instances have gone through testing and training. So that in a way help cater underfitting $\endgroup$ Jun 29 at 8:30
  • $\begingroup$ @MichaelM You are right but I am thinking in a way that cross-validation would help make model a little complex when you have less data and that way it somehow by a little margin reduces underfitting. $\endgroup$ Jun 29 at 8:34
  • $\begingroup$ @Ati : so your question is not about the fact that cross validation could reduce underfitting, but about the fact that changing the splits for cross validation coud reduce underfitting .. is that right ? $\endgroup$ Jun 29 at 9:33

1 Answer 1

2
$\begingroup$

The short answer is No. Cross validation does not "reduce the effects of underfitting" — or overfitting, for that matter.

I agree with the comments that your question seems to miss the point a little. The purpose of validation is to evaluate model performance after fitting, not to make the model more or less fit. It's the model's parameters that control its fitness, and tools like regularization can help you control its complexity. Validation (CV or holdout) might help you detect under- or over-fitness, but it doesn't improve or avoid either one.

I strongly recommend reading Raschka 2018, it's really the best thing that's been written about validation and it might help you figure out the best strategy for your task.

$\endgroup$
6
  • $\begingroup$ Thanks. So, is there by any chance I can say that if somehow it obviously won't reduce (as I didn't mean to say the model itself) after fitting but show the results in a way that we can say it catered the underfitting at validation time a bit even by a little margin? I am saying that cuz I have in my university course of ML been told that cross-fold reduces overfitting obviously at validation time, not the model itself. So I was asked the question of whether the cross-validation will reduce underfitting at validation time. So I gave the answer as yes and the reason I told is my question $\endgroup$ Jun 29 at 19:50
  • $\begingroup$ I'm sorry, I don't really understand what you mean by "cross-fold reduces overfitting at validation time", but it sounds wrong to me. Maybe ask your professor or TA what was meant. $\endgroup$
    – kwinkunks
    Jun 30 at 0:50
  • $\begingroup$ so let's say that cross-validation measures the performance when we apply cross-validation to our model it would give better results which seem that it catered the underfitting whereas if I use train-test-split it won't give better results the reason for that would be it doesn't train and test on every point. we just split it into train and test to a specific number. let me know if this makes any sense $\endgroup$ Jun 30 at 4:48
  • $\begingroup$ Changes in score resulting from changing the splits in your data will show you the variance between datasets (e.g. resulting from the splits having different distributions of features or labels). If the variance is high, your model is prone to overfitting and you need to address this by changing its hyperparameters, e.g. increasing regularization. Remember that after model selection, you are going to fit your model to all of your data, so the splits will have no bearing on the final model. $\endgroup$
    – kwinkunks
    Jun 30 at 14:01
  • $\begingroup$ So this increase in accuracy cant be considered the reduction of underfitting? $\endgroup$ Jul 1 at 11:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.