I am playing with the IRIS dataset and want to see underfitting and overfitting in action. I am using a multilayer perceptron (2 layers).
It is pretty easy to underfit (see the plot below), but I am having problems with overfitting. The dataset capacity is 600 (# of samples (150) times # of features (4)), so I should be able to overfit using a network with a capacity bigger than that. I am trying to use a multilayer perceptron with a total # of parameters of ~32000, but overfitting does not happen. What is going on? Thank you!
If I make a learning rate smaller everything get smoother, but still no overfitting.