1
$\begingroup$

I am working on a regression problem where I try to predict an Error with a NN with as little calculation steps as possible. Currently I have an input layer consisting of 21 Neurons and a Dense Output layer consisting of 4 Neurons. Would it make sense to use a Dropout Layer to prevent Overfitting? I read that using dropout layers makes sense in general but for this particular problem it seems counterintuitive, since I am creating an Error every time I ignore a certain Neuron so the other neurons would be trained to "overcompensate" this error.

$\endgroup$
  • 2
    $\begingroup$ Thank you very much This sentence: "Dropout is one of a bag of tricks used to train Deep Neural Networks. Your network is anything but deep, and it has a really small number of neurons." was exactly what I was looking for. If you post it as an answer, I will accept it :) $\endgroup$ – Philipps Aug 14 '18 at 20:28
4
$\begingroup$

Dropout is part of the array of techniques we developed to be able to train Deep Neural Networks on vast amount of data, without incurring in vanishing or exploding gradients: minibatch training, SGD, skip connections, batch normalization, ReLU units (though the jury is still out on these last ones: maybe they help with "pruning" the network, but not with vanishing gradients). They complement the "old school regularizarion tools" such as weight decay and early stopping.

Your neural network has only two layers (and, unless it has some nonlinear activation units, it's actually equivalent to a single layer NN): first layer has 21 units and second has 4. It's definitely not a Deep Neural Network...Now, without having more details on your specific application and data, I can't say for sure, but with such a shallow network it seems really unlikely that you will need dropout.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.