Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Dropout is a technique to reduce overfitting during the training phase of a neural network. DO NOT use this tag for dropout as in censoring or missing data in survival analysis or longitudinal data analysis.

3
votes
1answer
21 views

Why does dropout increase the training time per epoch in a neural network?

I'm training an MLP neural network with one hidden layer and batch gradient descent using Keras/Tensorflow. Applying dropout to the input layer increased the training time per epoch by about 25 %, ...
1
vote
1answer
64 views

Price Forecasting Problem

I am working on a project for price movement forecasting and I am stuck with poor quality predictions. At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the ...
0
votes
1answer
20 views

School Dropout Prediction

I have a dataset composed by several features group by some factors (academic, personal, economic). I would like to predict the risk (high, medium, low) of dropout and its respective risk percentage. ...
0
votes
0answers
25 views

Best practices to apply Layer normalization in recurrent networks

I'm trying to add layer normalization (in the encoder-level) to the Listen-attend-and-spell model for speech recognition tasks. To do so, I have done many experiments (all of them failed) to make my ...
0
votes
1answer
27 views

Dropout in Deep Neural Networks

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability p and not ...
1
vote
0answers
16 views

Training an ANN further once it reaches 100 % accuracy on training set

I have a very simply question: Does it make sense to further train an ANN once it reaches an accuracy of 100 % on the training data? I'm facing a binary classification problem and read this article ...
1
vote
0answers
24 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
2
votes
1answer
100 views

Drop-out as a multiplicative noise in deep neural networks

I am reading Ian Goodfellow's deep learning book, and I cannot understand the following lines: Another important aspect of dropout is that the noise is multiplicative. If the noise were additive ...
1
vote
1answer
77 views

Does it make sense to use a dropout layer in a neural network for a regression to predict an absolute Error?

I am working on a regression problem where I try to predict an Error with a NN with as little calculation steps as possible. Currently I have an input layer consisting of 21 Neurons and a Dense Output ...
0
votes
0answers
25 views

Why should each layer's child network output be close to parent network's output for variance regularizer?

I am reading up on PEA (Pseudo ensemble agreement) regularizer. specificaly in the neural networks domain. It introduces the concept of perturbing the model a little and forcing the model to make ...
1
vote
1answer
37 views

Why is dropout causing my network to overfit so badly? [closed]

I've been experimenting with various simple neural networks to test their performance. When I use the following architecture, I'm getting some very bad test error, which looks like overfitting. $$\...
1
vote
0answers
33 views

Why is dropout a good fit for ReLU units?

I have read this before that dropout interacts well with ReLU and recently going through SNN paper i came acoss this again. to quote: Dropout fits well to rectified linear units, since zero is in the ...
0
votes
1answer
17 views

Value of a dropout neuron with bias

I'm implementing dropout using the paper by Srivastava et al. In it, the authors suggest that when a neuron is dropped out, it is temporarily removed from the network completely: However, when ...
1
vote
1answer
803 views

Deep Learning : Using dropout in Autoencoders?

I am working with autoencoders and have few confusions, I am trying different autoencoders like: fully_connected autoencoder convolutional autoencoder denoising autoencoder I have two datasets, one ...
0
votes
1answer
30 views

Dropout: when do we eliminate neurons?

As you know at the training step according to the Dropout technique we eliminate every neuron with probability $p$. The question is do we eliminate neurons on every training instance (or batch) (or ...
0
votes
0answers
27 views

Back-propagation in presence of dropout [duplicate]

Intuitively dropout make sense to me but I don't understand how backpropagation works in presence of dropout. It looks like at each training step we backpropagate gradients to parameters in the ...
1
vote
0answers
140 views

Dropout: scaling the activation versus inverting the alpha dropout

I have the same question as in this post: Dropout: scaling the activation versus inverting the dropout but for alpha dropouts: I would like to know if I need to apply the scale factor of $p$ when ...
1
vote
1answer
65 views

Why do we need dropout in deep networks?

I have read some general statements about the usefulness of Dropout but the issue is still very vague to me. It is always said that it prohibits co-adaptability of neurons, but why it should be a bad ...
0
votes
1answer
32 views

Dropout and calculating values

I have been reading about dropout and understand what happens when training the network but I don't understand how it would work in calculating the network. Can someone please explain how the process ...
1
vote
0answers
29 views

Understanding dropout method: one bask per batch, or more?

I was talking with someone I know about the dropout method, and I realized we had different conceptions of how it worked. My impression was that there is one mask sampled per minibatch. His impression ...
1
vote
1answer
162 views

Weight scale inference when using dropout

Suppose I'm using dropout, and at test time I decide to do "weight scaling inference" (the method of predicting using the full network with weights multiplied by $p$, where $p$ is the probability of ...
0
votes
1answer
125 views

Fast dropout: How to compute the mean and variance of the approximating Gaussian?

In the Fast Dropout article by Wang and Manning, they talk about approximating the input to a hidden layer by a Gaussian with the same mean and variance as the inputs to the layer (see page 5). But ...
0
votes
1answer
34 views

Dropout for LSTMs

I've been just told that using Dropout For LSTMs is not considered the right thing these days. Is it true? If yes, what is recommended for overfitting prevention with LSTMs? Thanks!
3
votes
2answers
3k views

Dropout before Batch Normalization?

In the last course of the Deep Learning Specialization on Coursera from Andrew Ng, you can see that he uses the following sequence of layers on the output of an LSTM layer: Dropout -> BatchNorm -> ...
2
votes
2answers
108 views

Deep learning high dropout causes high model confidence scores

I am training an NLP classifier that maps input sentences to 1 of 50 categories. The model is a CNN language model, in which each input example is a 2d tensor of sentence length by word embedding ...
5
votes
1answer
2k views

Confused about Dropout implementation's in Tensorflow

I have a network whose input size is 100 and output size 2. Only these layers. I applied a dropout with keep_prob of 0.8 and I tried to understand the outcome. As expected, the dropout mask has ...
1
vote
1answer
426 views

Value of the keep probability when calculating loss with dropout

I'm training a small neural network (2 hidden layers) to classify the mnist images, and want to apply dropout regularization before my output layer. My first question: is it worth applying dropout ...
1
vote
1answer
851 views

Does it make sense to use dropout in last layer of regression neural network? [closed]

I have a neural network that I constructed in keras that goes from a LSTM recurrent layer > dropout > flattened > dense layer of 1 unit. Does this make sense to ...
0
votes
1answer
60 views

High validation accuracy without scaling paramters when using dropout

I was training a CNN network on German traffic sign classifier data. The architecture was- 3 convolutional layers with intermediate max pooling concatenated outputs of layer 2 and layer 3 to feed to ...
0
votes
0answers
266 views

High Dropout Rate affect on accuracy

I am testing a EMNIST classification network with 100 hidden layers and dropout rate set to 0.5. I am achieving virtually no overfitting, however only 70% accuracy. I understand that this is due to ...
2
votes
2answers
445 views

How is the dropout method different than Random Forests?

I've come across something called a dropout method that involves setting a threshold parameter $p$ and then for each predictor in your training set, generate a uniform random number. If that uniform ...
1
vote
1answer
100 views

Should I apply dropout if learning on huge dataset?

I am training an LSTM neural network for nlm on a big dataset: the model has about 100M learnable parameters and the dataset consists of about 2G characters. Therefore it seems that overfitting ...
5
votes
2answers
954 views

Dropout, what if all the nodes are dropped

When implementing dropout (or dropconnect) - do you need to account for the case that every node in a layer is dropped? Even though this is a very small chance, what is the correct approach to take ...
3
votes
1answer
5k views

Dropout makes performance worse

I am playing with dropout since all state of the art results in machine learning seem to be using it (for example, see here). I am familiar with all the guidelines (train longer, increase capacity of ...
4
votes
1answer
2k views

Why accuracy gradually increase then suddenly drop with dropout

I am building an image classification network in tensorflow(several convolutional layers and fully connected layers, then softmax cross entropy, optimize using Adam with a learning rate of 1e-4). ...
3
votes
0answers
309 views

Dropout causes overfitting

I am trying to experiment with dropout in 2 layer NN on notMNIST dataset using TensorFlow (assignment 3 in Google Deep Learning Course on Udacity). But adding dropout causes fall in test accuracy and ...
3
votes
0answers
116 views

Why do Srivastava et al. claim that “the best” theoretical regularization technique involves all possible network parameter settings?

In the original paper on Dropout by Srivastava, Hinton, Krizhevsky et al. (2014), the authors make this claim in the introduction: With unlimited computation, the best way to "regularize" a fixed-...
7
votes
2answers
4k views

How is Spatial Dropout in 2D implemented?

This is with refernce to the paper Efficient Object Localization Using Convolutional Networks, and from what I understand the dropout is implemented in 2D. After reading the code from Keras on how ...
3
votes
2answers
306 views

Dropout effectiveness on small neural networks

I implemented dropout on my neural networks. I tried to train the neural network to act as the f(x) = sin(x) function. During normal backpropagation without dropout ...
2
votes
2answers
692 views

If I use dropout in a neural network and run it for a large number of steps, do I risk deleting all the units?

I'm still trying to understand dropout completely, but this is what think is happening so far: At each step there is a chance p of a unit being set to zero. If a rectified linear unit (ReLU) is used ...
7
votes
1answer
2k views

Alternatives to L1, L2 and Dropout generalization

I have the following setup for a Finance/Machine Learning research project at my university: I'm applying a (Deep) Neural Network (MLP) with the following structure in Keras/Theano to distinguish ...
3
votes
1answer
715 views

Is there something analogous to dropout for classification problems?

I have only heard about dropout being applied to training of neural networks. Could the same technique, in theory, be applied to any iterative ML algorithm? For example, in mini-batch training, each ...
4
votes
1answer
95 views

What is a neuron bank?

In the Dropout paper they refer to neuron banks on page 15: ... the activity a^i (x,y) of a neuron in bank i at position (x, y) in the topographic organization ... What is a neuron bank? ...
1
vote
0answers
984 views

Convolutional Neural Networks regularization

I'm working on CNN and I have a question about regularization. Max-norm constraints (a form of gradient clipping) apply a rescale of the weight vector that satisfies $|W|_2<c$, normally in the ...
1
vote
1answer
1k views

Purpose of scaling weights/states when using dropout in a neural network

In Goodfellow's Deep Learning book (http://www.deeplearningbook.org/contents/regularization.html 7.12) they state: Because we usually use an inclusion probability of 1/2, the weight scaling rule ...
1
vote
2answers
291 views

Neural Network, questions on DropOut process [duplicate]

DropOut is a good way to reduce over-fitting in Neural Network, and I found a article on DropOut. My question is for each epoch, we should randomly pick half of the neurons of the network, and set ...
9
votes
3answers
8k views

How to explain dropout regularization in simple terms?

If you have a half page to explain dropout, how would you proceed? Which is the rationale behind this technique?
4
votes
1answer
2k views

Deep Learning: Use L2 and Dropout Regularization Simultaneously?

Is there a theoretical basis against using both L2 and Dropout regularization simultaneously for training a deep neural network? They are both related but could they be complementary if used together?...
36
votes
4answers
18k views

Where should I place dropout layers in a neural network?

Is there any general guidelines on where to place dropout layers in a neural network?