Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network. DO NOT use this tag for dropout as in censoring or missing data in survival analysis or longitudinal data analysis.

1
vote
0answers
19 views

To remove neural-network units or to increase drop-out?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
1
vote
0answers
35 views

How to apply dropout in LSTMs?

Dropout in fully connected neural networks is simpl to visualize, by just 'dropping' connections between units with some probability set by hyperparamter p. However, how dropout works in recurrent ...
2
votes
0answers
20 views

Dropout: Redundant representation vs. breaking up co-adaptation

Dropout is commonly used to regularize NNs. On one hand Dropout forces the network to have a redundant representation, since due to the dropout the NN can not rely on specific neurons for the ...
3
votes
1answer
94 views

Why does dropout increase the training time per epoch in a neural network?

I'm training an MLP neural network with one hidden layer and batch gradient descent using Keras/Tensorflow. Applying dropout to the input layer increased the training time per epoch by about 25 %, ...
1
vote
1answer
64 views

Price Forecasting Problem

I am working on a project for price movement forecasting and I am stuck with poor quality predictions. At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the ...
0
votes
1answer
21 views

School Dropout Prediction

I have a dataset composed by several features group by some factors (academic, personal, economic). I would like to predict the risk (high, medium, low) of dropout and its respective risk percentage. ...
0
votes
1answer
34 views

Dropout in Deep Neural Networks

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability p and not ...
1
vote
0answers
17 views

Training an ANN further once it reaches 100 % accuracy on training set

I have a very simply question: Does it make sense to further train an ANN once it reaches an accuracy of 100 % on the training data? I'm facing a binary classification problem and read this article ...
1
vote
0answers
27 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
2
votes
1answer
125 views

Drop-out as a multiplicative noise in deep neural networks

I am reading Ian Goodfellow's deep learning book, and I cannot understand the following lines: Another important aspect of dropout is that the noise is multiplicative. If the noise were additive ...
1
vote
1answer
136 views

Does it make sense to use a dropout layer in a neural network for a regression to predict an absolute Error?

I am working on a regression problem where I try to predict an Error with a NN with as little calculation steps as possible. Currently I have an input layer consisting of 21 Neurons and a Dense Output ...
0
votes
0answers
30 views

Why should each layer's child network output be close to parent network's output for variance regularizer?

I am reading up on PEA (Pseudo ensemble agreement) regularizer. specificaly in the neural networks domain. It introduces the concept of perturbing the model a little and forcing the model to make ...
1
vote
1answer
44 views

Why is dropout causing my network to overfit so badly? [closed]

I've been experimenting with various simple neural networks to test their performance. When I use the following architecture, I'm getting some very bad test error, which looks like overfitting. $$\...
1
vote
0answers
40 views

Why is dropout a good fit for ReLU units?

I have read this before that dropout interacts well with ReLU and recently going through SNN paper i came acoss this again. to quote: Dropout fits well to rectified linear units, since zero is in the ...
0
votes
1answer
20 views

Value of a dropout neuron with bias

I'm implementing dropout using the paper by Srivastava et al. In it, the authors suggest that when a neuron is dropped out, it is temporarily removed from the network completely: However, when ...
1
vote
1answer
1k views

Deep Learning : Using dropout in Autoencoders?

I am working with autoencoders and have few confusions, I am trying different autoencoders like: fully_connected autoencoder convolutional autoencoder denoising autoencoder I have two datasets, one ...
0
votes
1answer
31 views

Dropout: when do we eliminate neurons?

As you know at the training step according to the Dropout technique we eliminate every neuron with probability $p$. The question is do we eliminate neurons on every training instance (or batch) (or ...
0
votes
0answers
27 views

Back-propagation in presence of dropout [duplicate]

Intuitively dropout make sense to me but I don't understand how backpropagation works in presence of dropout. It looks like at each training step we backpropagate gradients to parameters in the ...
1
vote
0answers
157 views

Dropout: scaling the activation versus inverting the alpha dropout

I have the same question as in this post: Dropout: scaling the activation versus inverting the dropout but for alpha dropouts: I would like to know if I need to apply the scale factor of $p$ when ...
1
vote
1answer
66 views

Why do we need dropout in deep networks?

I have read some general statements about the usefulness of Dropout but the issue is still very vague to me. It is always said that it prohibits co-adaptability of neurons, but why it should be a bad ...
0
votes
1answer
35 views

Dropout and calculating values

I have been reading about dropout and understand what happens when training the network but I don't understand how it would work in calculating the network. Can someone please explain how the process ...
1
vote
0answers
38 views

Understanding dropout method: one bask per batch, or more?

I was talking with someone I know about the dropout method, and I realized we had different conceptions of how it worked. My impression was that there is one mask sampled per minibatch. His impression ...
1
vote
1answer
237 views

Weight scale inference when using dropout

Suppose I'm using dropout, and at test time I decide to do "weight scaling inference" (the method of predicting using the full network with weights multiplied by $p$, where $p$ is the probability of ...
0
votes
1answer
158 views

Fast dropout: How to compute the mean and variance of the approximating Gaussian?

In the Fast Dropout article by Wang and Manning, they talk about approximating the input to a hidden layer by a Gaussian with the same mean and variance as the inputs to the layer (see page 5). But ...
0
votes
1answer
38 views

Dropout for LSTMs

I've been just told that using Dropout For LSTMs is not considered the right thing these days. Is it true? If yes, what is recommended for overfitting prevention with LSTMs? Thanks!
3
votes
2answers
4k views

Dropout before Batch Normalization?

In the last course of the Deep Learning Specialization on Coursera from Andrew Ng, you can see that he uses the following sequence of layers on the output of an LSTM layer: Dropout -> BatchNorm -> ...
2
votes
2answers
149 views

Deep learning high dropout causes high model confidence scores

I am training an NLP classifier that maps input sentences to 1 of 50 categories. The model is a CNN language model, in which each input example is a 2d tensor of sentence length by word embedding ...
5
votes
1answer
3k views

Confused about Dropout implementation's in Tensorflow

I have a network whose input size is 100 and output size 2. Only these layers. I applied a dropout with keep_prob of 0.8 and I tried to understand the outcome. As expected, the dropout mask has ...
1
vote
1answer
526 views

Value of the keep probability when calculating loss with dropout

I'm training a small neural network (2 hidden layers) to classify the mnist images, and want to apply dropout regularization before my output layer. My first question: is it worth applying dropout ...
2
votes
1answer
981 views

Does it make sense to use dropout in last layer of regression neural network? [closed]

I have a neural network that I constructed in keras that goes from a LSTM recurrent layer > dropout > flattened > dense layer of 1 unit. Does this make sense to ...
0
votes
1answer
64 views

High validation accuracy without scaling paramters when using dropout

I was training a CNN network on German traffic sign classifier data. The architecture was- 3 convolutional layers with intermediate max pooling concatenated outputs of layer 2 and layer 3 to feed to ...
2
votes
2answers
520 views

How is the dropout method different than Random Forests?

I've come across something called a dropout method that involves setting a threshold parameter $p$ and then for each predictor in your training set, generate a uniform random number. If that uniform ...
1
vote
1answer
106 views

Should I apply dropout if learning on huge dataset?

I am training an LSTM neural network for nlm on a big dataset: the model has about 100M learnable parameters and the dataset consists of about 2G characters. Therefore it seems that overfitting ...
5
votes
2answers
1k views

Dropout, what if all the nodes are dropped

When implementing dropout (or dropconnect) - do you need to account for the case that every node in a layer is dropped? Even though this is a very small chance, what is the correct approach to take ...
4
votes
1answer
7k views

Dropout makes performance worse

I am playing with dropout since all state of the art results in machine learning seem to be using it (for example, see here). I am familiar with all the guidelines (train longer, increase capacity of ...
4
votes
2answers
2k views

Why accuracy gradually increase then suddenly drop with dropout

I am building an image classification network in tensorflow(several convolutional layers and fully connected layers, then softmax cross entropy, optimize using Adam with a learning rate of 1e-4). ...
3
votes
0answers
333 views

Dropout causes overfitting

I am trying to experiment with dropout in 2 layer NN on notMNIST dataset using TensorFlow (assignment 3 in Google Deep Learning Course on Udacity). But adding dropout causes fall in test accuracy and ...
3
votes
0answers
118 views

Why do Srivastava et al. claim that “the best” theoretical regularization technique involves all possible network parameter settings?

In the original paper on Dropout by Srivastava, Hinton, Krizhevsky et al. (2014), the authors make this claim in the introduction: With unlimited computation, the best way to "regularize" a fixed-...
9
votes
2answers
5k views

How is Spatial Dropout in 2D implemented?

This is with refernce to the paper Efficient Object Localization Using Convolutional Networks, and from what I understand the dropout is implemented in 2D. After reading the code from Keras on how ...
3
votes
2answers
357 views

Dropout effectiveness on small neural networks

I implemented dropout on my neural networks. I tried to train the neural network to act as the f(x) = sin(x) function. During normal backpropagation without dropout ...
2
votes
2answers
770 views

If I use dropout in a neural network and run it for a large number of steps, do I risk deleting all the units?

I'm still trying to understand dropout completely, but this is what think is happening so far: At each step there is a chance p of a unit being set to zero. If a rectified linear unit (ReLU) is used ...
7
votes
1answer
2k views

Alternatives to L1, L2 and Dropout generalization

I have the following setup for a Finance/Machine Learning research project at my university: I'm applying a (Deep) Neural Network (MLP) with the following structure in Keras/Theano to distinguish ...
3
votes
1answer
765 views

Is there something analogous to dropout for classification problems?

I have only heard about dropout being applied to training of neural networks. Could the same technique, in theory, be applied to any iterative ML algorithm? For example, in mini-batch training, each ...
4
votes
1answer
96 views

What is a neuron bank?

In the Dropout paper they refer to neuron banks on page 15: ... the activity a^i (x,y) of a neuron in bank i at position (x, y) in the topographic organization ... What is a neuron bank? ...
1
vote
0answers
1k views

Convolutional Neural Networks regularization

I'm working on CNN and I have a question about regularization. Max-norm constraints (a form of gradient clipping) apply a rescale of the weight vector that satisfies $|W|_2<c$, normally in the ...
1
vote
1answer
2k views

Purpose of scaling weights/states when using dropout in a neural network

In Goodfellow's Deep Learning book (http://www.deeplearningbook.org/contents/regularization.html 7.12) they state: Because we usually use an inclusion probability of 1/2, the weight scaling rule ...
1
vote
2answers
336 views

Neural Network, questions on DropOut process [duplicate]

DropOut is a good way to reduce over-fitting in Neural Network, and I found a article on DropOut. My question is for each epoch, we should randomly pick half of the neurons of the network, and set ...
11
votes
3answers
9k views

How to explain dropout regularization in simple terms?

If you have a half page to explain dropout, how would you proceed? Which is the rationale behind this technique?
4
votes
1answer
2k views

Deep Learning: Use L2 and Dropout Regularization Simultaneously?

Is there a theoretical basis against using both L2 and Dropout regularization simultaneously for training a deep neural network? They are both related but could they be complementary if used together?...