I am currently having a problem with my network, it either overfits with the training data so i tried to solve the problem by applying dropout layers to all my hidden layers. This in turn made it so my network is now underfitting and never getting all the neurons passed a MSE of 0.1. The network itself isn't that big with a brief description below.
My question is how would i solve this problem of underfitting and overfitting?
Network structure: 100 -> 30 -> 30 -> 3
I am working with wifi signals and the input value is equal to the strength. The input for the network coresponds to the signal strength of my given routers. The input values would be the following:
1: good signal
0: bad signal
-1: no signal
The following picture shows the training for the network without dropout neurons (overfitting) MSE of single
is the MSE for one specific training value from my training set.
Results of running testing data within the network:
Same Structured network but with dropout at 5%:
Both were running for 100000 iterations. I am using stochastic gradient descent with a batch size of 15 training values each iteration and a total amount of training data of 240.
Function being used tanh.
Dropout probability used: 10%, 20%, 50%
Network Sizes tried before:
- 100, 10, 3
- 100, 50 ,25, 10 ,3
- 100, 100, 100, 3
- 100, 12, 3 = best so far
- 100 , 3
I just do not get a good result on my testing data. I would be very happy if someone could help me with this to increase the accuracy of my network on the testing data.