In fitting a neural network, initial validation and learning curves seemed to indicate my model was overfitted. After trying a few options to decrease overfitting that had no impact, I'm questioning if my model is in fact overfitted.
Why I reached conclusion of overfitting
I performed validation using 3 runs of 10-fold CV to optimize the number of neurons in the single hidden layer of my NN.
This indicated to me that lower number of neurons is just as effective as higher. Because my test set consistently performs worse than my train set, I also interpreted overfitting, though performance for both train and test is rather low.
Then I used learning curve, again with 10-fold CV to assess impact of data set sizes:
This still indicated overfitting since test set is lower than train set, but also suggested more data may help since curves (likely) haven't converged.
Reducing overfitting with less training time and regularization
I first tried reducing the maximum number of epochs, using validation curve with 3 runs of 10-fold CV.
This led me to the conclusion that reducing number of max epochs wouldn't really reduce overfitting, I selected optimal epochs at 800 where the curves level off.
Then, I tried L2 regularization via
sklearn's $\alpha$ parameter.
Regularization seems to have no effect.
The fact that my efforts in reducing training time and increasing regularization penalty did nothing is making me call into question whether my model is in fact overfitted.
1. I'd appreciate some input on whether my determination of overfitting is correct. I'm unclear if my regularization results are telling me my conclusion of overfitting is off.
2. If my determination of overfitting is correct, might there be a reason why less training/regularization had no effect? Any other options I might try?