0
$\begingroup$

In fitting a neural network, initial validation and learning curves seemed to indicate my model was overfitted. After trying a few options to decrease overfitting that had no impact, I'm questioning if my model is in fact overfitted.

Why I reached conclusion of overfitting

I performed validation using 3 runs of 10-fold CV to optimize the number of neurons in the single hidden layer of my NN.

enter image description here

This indicated to me that lower number of neurons is just as effective as higher. Because my test set consistently performs worse than my train set, I also interpreted overfitting, though performance for both train and test is rather low.

Then I used learning curve, again with 10-fold CV to assess impact of data set sizes:

enter image description here

This still indicated overfitting since test set is lower than train set, but also suggested more data may help since curves (likely) haven't converged.

Reducing overfitting with less training time and regularization

I first tried reducing the maximum number of epochs, using validation curve with 3 runs of 10-fold CV.

enter image description here

This led me to the conclusion that reducing number of max epochs wouldn't really reduce overfitting, I selected optimal epochs at 800 where the curves level off.

Then, I tried L2 regularization via sklearn's $\alpha$ parameter.

enter image description here

Regularization seems to have no effect.

Question

The fact that my efforts in reducing training time and increasing regularization penalty did nothing is making me call into question whether my model is in fact overfitted.

1. I'd appreciate some input on whether my determination of overfitting is correct. I'm unclear if my regularization results are telling me my conclusion of overfitting is off.

2. If my determination of overfitting is correct, might there be a reason why less training/regularization had no effect? Any other options I might try?

Improve this question
$\endgroup$
2
  • $\begingroup$ Did you look into actual error? Can it be that the model performs bad on the training set and even worse on the test set? Generally, however, discrepancies in the behavior on training and test data indicate overfitting. $\endgroup$
    – Jan Kukacka
    Nov 18, 2019 at 12:34
  • $\begingroup$ Your train and test performance start diverging after about ~10 epochs, where test performance seems to reach its plateau. How many observations do you have and how large is your batch size, if you train using stochastic gradient descent with mini-batches? Your number of model updates might be large, even after few epochs. You could try Dropout as an alternative way to reduce overfitting. $\endgroup$
    – Johannes
    Nov 18, 2019 at 13:56

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.