2
$\begingroup$

So I want to know whether or not my models are overfitting or the difference between train and validation errors are decent.

$L$: is the number of neighbors

The first column is the train error

The second column is the validation error

enter image description here

Okay. Actually I got those values from 5-fold cross validation. So the column error on the right is the averaged test error over all folds and the errors on the column on the left are the averaged train error over all folds. From what i know you are supposed to know if a model is overfitting or not by adding more data to the training set? In this example all I did was to sweep the values of the $L$ neighbors.

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ What metric exactly are we looking at here? $\endgroup$ – user2974951 Sep 9 '19 at 7:22
  • $\begingroup$ We are looking at RMSE $\endgroup$ – Delan Sep 9 '19 at 7:29
  • 1
    $\begingroup$ There is something weird going on, since the validation error remains pretty much constant throughout the L's while the training error actually increases. If you were overfitting you would expect this to decrease not increase. $\endgroup$ – user2974951 Sep 9 '19 at 7:35
  • $\begingroup$ Okay. Actually I got those values from 5-fold cross validation. So the column error on the right is the averaged test error over all folds and the errors on the column on the left are the averaged train error over all folds. From what i know you are supposed to know if a model is overfitting or not by adding more data to the training set? In this example all I did was to sweep the values of the L neighbors. $\endgroup$ – Delan Sep 9 '19 at 7:39
  • 1
    $\begingroup$ Still, the results are telling you that you are getting worse results on your training set if you increase L, while the validation results tell you that you are not really learning anything new if you increase L, you are getting very similar results across the L's. $\endgroup$ – user2974951 Sep 9 '19 at 7:42
1
$\begingroup$

Not sure if this will answer your question, but look at it this way:

In OLS, the more regressors ("features") you put into your model, the more likely you are to overfit the model.

But in the case of k-nearest neighbors (as I assume you're doing), increasing the number of neighbors ($L$) would actually lead to underfitting (see here: Does k-NN with k=1 always implies overfitting?)

So this is why I suggested lowering $L$. You should see train error go down, and test error go (generally) up. You could then decide what your $L$ should be, based on where the errors are "balanced".

Visually, you're on the left side of this graph. You need to explore further to the right (in the direction of decreasing bias) and this could be done by lowering $L$.

enter image description here

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer! It helps me a lot! However, my question is sort of tricky because those train and validation errors were obtained from 5 fold cross validation. Whenever you apply that technique, you always get sticked to the lowest cross validation error and forget about the training one. I’m using cross validation for tunning hyperparameters purposes. Once the lowest cross validation error is obtained, throw the best parameters’ combination, I retrain the model on the entire dataset and test on the test set. From there on, I think I could apply your approach. $\endgroup$ – Delan Sep 9 '19 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.