I am working on a regression problem where I try to predict an Error with a NN with as little calculation steps as possible. Currently I have an input layer consisting of 21 Neurons and a Dense Output layer consisting of 4 Neurons. Would it make sense to use a Dropout Layer to prevent Overfitting? I read that using dropout layers makes sense in general but for this particular problem it seems counterintuitive, since I am creating an Error every time I ignore a certain Neuron so the other neurons would be trained to "overcompensate" this error.

  • 2
    $\begingroup$ Thank you very much This sentence: "Dropout is one of a bag of tricks used to train Deep Neural Networks. Your network is anything but deep, and it has a really small number of neurons." was exactly what I was looking for. If you post it as an answer, I will accept it :) $\endgroup$
    – Philipps
    Commented Aug 14, 2018 at 20:28
  • 2
    $\begingroup$ The larger question of "Does it make sense to use a dropout layer in a neural network for a regression to predict an absolute Error?" still remains unanswered though. A one word answer would be No. Dropout works well with relative scales (e.g. in case of softmax etc where the relative values are all that matter) and not on an absolute scale due to instability because of difference introduced between train and test process $\endgroup$
    – Allohvk
    Commented Aug 16, 2021 at 6:18

1 Answer 1


Dropout is part of the array of techniques we developed to be able to train Deep Neural Networks on vast amount of data, without incurring in vanishing or exploding gradients: minibatch training, SGD, skip connections, batch normalization, ReLU units (though the jury is still out on these last ones: maybe they help with "pruning" the network, but not with vanishing gradients). They complement the "old school regularizarion tools" such as weight decay and early stopping.

Your neural network has only two layers (and, unless it has some nonlinear activation units, it's actually equivalent to a single layer NN): first layer has 21 units and second has 4. It's definitely not a Deep Neural Network...Now, without having more details on your specific application and data, I can't say for sure, but with such a shallow network it seems really unlikely that you will need dropout.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.