0
$\begingroup$

I am currently having a problem with my network, it either overfits with the training data so i tried to solve the problem by applying dropout layers to all my hidden layers. This in turn made it so my network is now underfitting and never getting all the neurons passed a MSE of 0.1. The network itself isn't that big with a brief description below.

My question is how would i solve this problem of underfitting and overfitting?

Network structure: 100 -> 30 -> 30 -> 3

I am working with wifi signals and the input value is equal to the strength. The input for the network coresponds to the signal strength of my given routers. The input values would be the following:

1: good signal
0: bad signal
-1: no signal

The following picture shows the training for the network without dropout neurons (overfitting) MSE of single is the MSE for one specific training value from my training set.

enter image description here

Results of running testing data within the network:

enter image description here

Same Structured network but with dropout at 5%:

enter image description here

Both were running for 100000 iterations. I am using stochastic gradient descent with a batch size of 15 training values each iteration and a total amount of training data of 240. 

Function being used tanh.

Dropout probability used: 10%, 20%, 50%

Network Sizes tried before:

 - 100, 10, 3 
 - 100, 50 ,25, 10 ,3
 - 100, 100, 100, 3
 - 100, 12, 3 = best so far
 - 100 , 3

I just do not get a good result on my testing data. I would be very happy if someone could help me with this to increase the accuracy of my network on the testing data.

Improve this question
$\endgroup$
13
  • $\begingroup$ The fact that you applied dropout doesn't really tell much unless you specify dropout probability. Did you try other dropout probabilities? Less layers? $\endgroup$
    – Jakub Bartczuk
    Apr 2, 2018 at 11:00
  • $\begingroup$ @JakubBartczuk i added some network sizes i have added below and dropout probability's i have tried the dropout probability's all result in the same MSE getting to 1.... E-4 and no higher $\endgroup$
    – Darren
    Apr 2, 2018 at 11:07
  • $\begingroup$ I don't understand. You previously wrote that your MSE is 0.1. MSE of 1E-4 is way better $\endgroup$
    – Jakub Bartczuk
    Apr 2, 2018 at 11:09
  • $\begingroup$ BTW you can always try to use smaller dropout. $\endgroup$
    – Jakub Bartczuk
    Apr 2, 2018 at 11:09
  • $\begingroup$ @JakubBartczuk i trained it overnight for around 10000000 iterations (stopped it when i woke up but it was just over that) and got it that low. I will try 2% dropout for a few hours and tell you how it goes. $\endgroup$
    – Darren
    Apr 2, 2018 at 11:12

1 Answer 1

Reset to default
1
$\begingroup$

Don’t know if you have specific design issues for your problem, but your training data size of 240 is usually too small to train an accurate model for 100-dimension input. You may easily get either overfitting if your network consists of several layers - with a lot more parameters than samples, or underfitting with too shallow network - with inadequate capacity and/or nonlinearity to model your problem. Neither of them can bring you good test result - as a measure of your model’s generalization.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.