5
$\begingroup$

I am playing with the IRIS dataset and want to see underfitting and overfitting in action. I am using a multilayer perceptron (2 layers).

It is pretty easy to underfit (see the plot below), but I am having problems with overfitting. The dataset capacity is 600 (# of samples (150) times # of features (4)), so I should be able to overfit using a network with a capacity bigger than that. I am trying to use a multilayer perceptron with a total # of parameters of ~32000, but overfitting does not happen. What is going on? Thank you!

Val and train

Val only

enter image description here

If I make a learning rate smaller everything get smoother, but still no overfitting.

enter image description here

Improve this question
$\endgroup$
11
  • $\begingroup$ Could you repost the plot, excluding the training lines? $\endgroup$
    – jbowman
    Aug 24, 2017 at 22:08
  • 1
    $\begingroup$ For this type of analysis, you should be comparing a proper loss function like cross-entropy. Accuracy is necessarily limited to the set from 0 to 1 in fractions of $1/n$, so it will conceal considerable information about what's going on. $\endgroup$
    – Sycorax ♦
    Aug 24, 2017 at 22:13
  • $\begingroup$ @jbowman Sure, I added another plot. $\endgroup$
    – Yuri
    Aug 24, 2017 at 22:13
  • $\begingroup$ Notice how, when you get out to epochs 50+, the purple line seems to give the poorest average result, followed by the turquoise (or whatever color that is) line, then it looks like the red is the next worst, which leaves the green as the best. @Sycorax is exactly correct, you should be using a proper loss function, but even with this accuracy measure you can, in an informal way, see a pattern there. $\endgroup$
    – jbowman
    Aug 24, 2017 at 22:16
  • $\begingroup$ @jbowman But those are just some fluctuations, in the end all lines (except the blue one) are pretty close to each other. $\endgroup$
    – Yuri
    Aug 24, 2017 at 22:19

1 Answer 1

Reset to default
4
$\begingroup$

Are you measuring training set performance there? It is per definition impossible to see overfit in the training set performance (underfit you would see).

Also, it is in theory absolutely possible to have a data-set on which you cannot overfit. The classes could be so well separated that even a horribly overfitting algorithm like 1NN always finds a one nearest neighbor that is of the same class.

With the iris data-set. Setosa is very easy to separate from the other two, but virginica and versicolor have some overlap so that you should see decreasing test-set performance as you overfit. My best guess is that you are measuring training set performance.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Ofc I measure both training and testing performances, dashed lines - accuracy on training set. I added the label. But I like the second thought though. $\endgroup$
    – Yuri
    Aug 24, 2017 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.