I am very new to neural networks. I am running a simple ANN with Keras (TensorFlow backend) on a dataset with around 5000 observations and 4 features.
I am trying different parameters and such things in the neural network, and then plotting the train and test error. For instance I increased/decreased batch_size, I increased/decreased the number of epochs, I increased/decreased the amount of data I used, I increased/decreased the number of nodes in the first and only hidden layer, I increased/decreased the number of hidden layers and so on.
Plotting the training and test error suggests that some of them may lead to overfitting. So here is my question:
Which of these features of a NN COULD lead to overfitting? And which ones am I missing? [in general!]
- Increasing/Decreasing batch size.
- Increasing/Decreasing epoch size
- Increasing/Decreasing number of neurons in first hidden layer (or other layers).
- Increasing/Decreasing the number of hidden layers
My guess is that increasing the number of neurons and layers can lead to overfitting. Increasing the number of epochs doesn't, while increasing the batchsize could lead to overfitting.
Is this correct? Independently of the number of samples we have, which of these COULD lead to overfitting? And why?
Furthermore, which other features of a neural network could lead to overfitting?