Neural Network overfitting and underfitting problem for wifi triangulation

I am currently having a problem with my network, it either overfits with the training data so i tried to solve the problem by applying dropout layers to all my hidden layers. This in turn made it so my network is now underfitting and never getting all the neurons passed a MSE of 0.1. The network itself isn't that big with a brief description below.

My question is how would i solve this problem of underfitting and overfitting?

Network structure: 100 -> 30 -> 30 -> 3

I am working with wifi signals and the input value is equal to the strength. The input for the network coresponds to the signal strength of my given routers. The input values would be the following:

1: good signal
-1: no signal


The following picture shows the training for the network without dropout neurons (overfitting) MSE of single is the MSE for one specific training value from my training set.

Results of running testing data within the network:

Same Structured network but with dropout at 5%:

Both were running for 100000 iterations. I am using stochastic gradient descent with a batch size of 15 training values each iteration and a total amount of training data of 240.

Function being used tanh.

Dropout probability used: 10%, 20%, 50%

Network Sizes tried before:

- 100, 10, 3
- 100, 50 ,25, 10 ,3
- 100, 100, 100, 3
- 100, 12, 3 = best so far
- 100 , 3


I just do not get a good result on my testing data. I would be very happy if someone could help me with this to increase the accuracy of my network on the testing data.

• The fact that you applied dropout doesn't really tell much unless you specify dropout probability. Did you try other dropout probabilities? Less layers? Apr 2, 2018 at 11:00
• @JakubBartczuk i added some network sizes i have added below and dropout probability's i have tried the dropout probability's all result in the same MSE getting to 1.... E-4 and no higher Apr 2, 2018 at 11:07
• I don't understand. You previously wrote that your MSE is 0.1. MSE of 1E-4 is way better Apr 2, 2018 at 11:09
• BTW you can always try to use smaller dropout. Apr 2, 2018 at 11:09
• @JakubBartczuk i trained it overnight for around 10000000 iterations (stopped it when i woke up but it was just over that) and got it that low. I will try 2% dropout for a few hours and tell you how it goes. Apr 2, 2018 at 11:12