Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network. DO NOT use this tag for dropout as in censoring or missing data in survival analysis or longitudinal data analysis.

4
votes
2answers
207 views

Creating thinned models in during Dropout process

Applying dropout to a neural network amounts to sampling a “thinned” network from it. The thinned network consists of all the units that survived dropout. A neural net with n units can be seen as a ...
7
votes
1answer
198 views

Dropout in Linear Regression

I've been reading the original paper on dropout, (https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) and in the linear regression section, it is stated that: $\mathbb{E}_{R\sim Bernoulli(p)}\...
0
votes
2answers
30 views

How many neurons are actually dropped when using dropout?

I understand that when using dropout, a single neuron can be described using Bernoulli random variable and for a set of neurons it can be described as Binomial random variable When using Dropout, we ...
0
votes
0answers
29 views

How dose dropout affect Weights and Bias?

I applied dropout in my network , and it worked , but i can't interpret dropout effects on weight and bias, to be more specific , i can't interpret why appling droput and not applying dropout have a ...
0
votes
1answer
43 views

Theoretical foundation for dropout in neural networks

Can someone point me to a thorough theoretical foundation for dropout in training neural networks? So far I have found only handwaving explanations (e.g. Goodfellow's textbook and the original paper) ...
2
votes
1answer
82 views

Where to include Dropout in stacked autoencoder

I'm using Keras to implement a stacked autoencoder, and I think it may be overfitting. I wanted to include dropout, and keep reading about the use of dropout in autoencoders, but I cannot find any ...
0
votes
0answers
11 views

What to do when the activation is non-linear when rescaling to compensate for dropout?

At 54:24 of this video, it says that once there is non-linearity in the activation function, the expectation is not exact. So would it post a problem for non-linear activation? Then how would you ...
1
vote
1answer
45 views

What does it mean by “approach the performance of the Bayesian gold standard”?

It is a sentence in Dropout paper(Dropout: A Simple Way to Prevent Neural Networks from Overfitting). "This can sometimes be approximated quite well for simple or small models, but we would like to ...
1
vote
0answers
21 views

To remove neural-network units or to increase drop-out?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
1
vote
0answers
41 views

How to apply dropout in LSTMs?

Dropout in fully connected neural networks is simpl to visualize, by just 'dropping' connections between units with some probability set by hyperparamter p. However, how dropout works in recurrent ...
2
votes
0answers
39 views

Dropout: Redundant representation vs. breaking up co-adaptation

Dropout is commonly used to regularize NNs. On one hand Dropout forces the network to have a redundant representation, since due to the dropout the NN can not rely on specific neurons for the ...
3
votes
1answer
246 views

Why does dropout increase the training time per epoch in a neural network?

I'm training an MLP neural network with one hidden layer and batch gradient descent using Keras/Tensorflow. Applying dropout to the input layer increased the training time per epoch by about 25 %, ...
1
vote
1answer
67 views

Price Forecasting Problem

I am working on a project for price movement forecasting and I am stuck with poor quality predictions. At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the ...
0
votes
1answer
25 views

School Dropout Prediction

I have a dataset composed by several features group by some factors (academic, personal, economic). I would like to predict the risk (high, medium, low) of dropout and its respective risk percentage. ...
0
votes
1answer
62 views

Dropout in Deep Neural Networks

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability p and not ...
1
vote
0answers
18 views

Training an ANN further once it reaches 100 % accuracy on training set

I have a very simply question: Does it make sense to further train an ANN once it reaches an accuracy of 100 % on the training data? I'm facing a binary classification problem and read this article ...
1
vote
0answers
39 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
2
votes
1answer
179 views

Drop-out as a multiplicative noise in deep neural networks

I am reading Ian Goodfellow's deep learning book, and I cannot understand the following lines: Another important aspect of dropout is that the noise is multiplicative. If the noise were additive ...
1
vote
1answer
219 views

Does it make sense to use a dropout layer in a neural network for a regression to predict an absolute Error?

I am working on a regression problem where I try to predict an Error with a NN with as little calculation steps as possible. Currently I have an input layer consisting of 21 Neurons and a Dense Output ...
0
votes
0answers
34 views

Why should each layer's child network output be close to parent network's output for variance regularizer?

I am reading up on PEA (Pseudo ensemble agreement) regularizer. specificaly in the neural networks domain. It introduces the concept of perturbing the model a little and forcing the model to make ...
1
vote
1answer
51 views

Why is dropout causing my network to overfit so badly? [closed]

I've been experimenting with various simple neural networks to test their performance. When I use the following architecture, I'm getting some very bad test error, which looks like overfitting. $$\...
1
vote
0answers
44 views

Why is dropout a good fit for ReLU units?

I have read this before that dropout interacts well with ReLU and recently going through SNN paper i came acoss this again. to quote: Dropout fits well to rectified linear units, since zero is in the ...
0
votes
1answer
25 views

Value of a dropout neuron with bias

I'm implementing dropout using the paper by Srivastava et al. In it, the authors suggest that when a neuron is dropped out, it is temporarily removed from the network completely: However, when ...
1
vote
1answer
2k views

Deep Learning : Using dropout in Autoencoders?

I am working with autoencoders and have few confusions, I am trying different autoencoders like: fully_connected autoencoder convolutional autoencoder denoising autoencoder I have two datasets, one ...
0
votes
1answer
33 views

Dropout: when do we eliminate neurons?

As you know at the training step according to the Dropout technique we eliminate every neuron with probability $p$. The question is do we eliminate neurons on every training instance (or batch) (or ...
0
votes
0answers
28 views

Back-propagation in presence of dropout [duplicate]

Intuitively dropout make sense to me but I don't understand how backpropagation works in presence of dropout. It looks like at each training step we backpropagate gradients to parameters in the ...
1
vote
0answers
188 views

Dropout: scaling the activation versus inverting the alpha dropout

I have the same question as in this post: Dropout: scaling the activation versus inverting the dropout but for alpha dropouts: I would like to know if I need to apply the scale factor of $p$ when ...
1
vote
1answer
67 views

Why do we need dropout in deep networks?

I have read some general statements about the usefulness of Dropout but the issue is still very vague to me. It is always said that it prohibits co-adaptability of neurons, but why it should be a bad ...
0
votes
1answer
38 views

Dropout and calculating values

I have been reading about dropout and understand what happens when training the network but I don't understand how it would work in calculating the network. Can someone please explain how the process ...
1
vote
0answers
56 views

Understanding dropout method: one bask per batch, or more?

I was talking with someone I know about the dropout method, and I realized we had different conceptions of how it worked. My impression was that there is one mask sampled per minibatch. His impression ...
1
vote
1answer
452 views

Weight scale inference when using dropout

Suppose I'm using dropout, and at test time I decide to do "weight scaling inference" (the method of predicting using the full network with weights multiplied by $p$, where $p$ is the probability of ...
0
votes
1answer
222 views

Fast dropout: How to compute the mean and variance of the approximating Gaussian?

In the Fast Dropout article by Wang and Manning, they talk about approximating the input to a hidden layer by a Gaussian with the same mean and variance as the inputs to the layer (see page 5). But ...
0
votes
1answer
41 views

Dropout for LSTMs

I've been just told that using Dropout For LSTMs is not considered the right thing these days. Is it true? If yes, what is recommended for overfitting prevention with LSTMs? Thanks!
3
votes
2answers
5k views

Dropout before Batch Normalization?

In the last course of the Deep Learning Specialization on Coursera from Andrew Ng, you can see that he uses the following sequence of layers on the output of an LSTM layer: Dropout -> BatchNorm -> ...
2
votes
2answers
230 views

Deep learning high dropout causes high model confidence scores

I am training an NLP classifier that maps input sentences to 1 of 50 categories. The model is a CNN language model, in which each input example is a 2d tensor of sentence length by word embedding ...
5
votes
1answer
3k views

Confused about Dropout implementation's in Tensorflow

I have a network whose input size is 100 and output size 2. Only these layers. I applied a dropout with keep_prob of 0.8 and I tried to understand the outcome. As expected, the dropout mask has ...
1
vote
1answer
648 views

Value of the keep probability when calculating loss with dropout

I'm training a small neural network (2 hidden layers) to classify the mnist images, and want to apply dropout regularization before my output layer. My first question: is it worth applying dropout ...
2
votes
1answer
1k views

Does it make sense to use dropout in last layer of regression neural network? [closed]

I have a neural network that I constructed in keras that goes from a LSTM recurrent layer > dropout > flattened > dense layer of 1 unit. Does this make sense to ...
0
votes
1answer
73 views

High validation accuracy without scaling paramters when using dropout

I was training a CNN network on German traffic sign classifier data. The architecture was- 3 convolutional layers with intermediate max pooling concatenated outputs of layer 2 and layer 3 to feed to ...
2
votes
2answers
599 views

How is the dropout method different than Random Forests?

I've come across something called a dropout method that involves setting a threshold parameter $p$ and then for each predictor in your training set, generate a uniform random number. If that uniform ...
1
vote
1answer
115 views

Should I apply dropout if learning on huge dataset?

I am training an LSTM neural network for nlm on a big dataset: the model has about 100M learnable parameters and the dataset consists of about 2G characters. Therefore it seems that overfitting ...
5
votes
2answers
1k views

Dropout, what if all the nodes are dropped

When implementing dropout (or dropconnect) - do you need to account for the case that every node in a layer is dropped? Even though this is a very small chance, what is the correct approach to take ...
4
votes
1answer
8k views

Dropout makes performance worse

I am playing with dropout since all state of the art results in machine learning seem to be using it (for example, see here). I am familiar with all the guidelines (train longer, increase capacity of ...
4
votes
2answers
2k views

Why accuracy gradually increase then suddenly drop with dropout

I am building an image classification network in tensorflow(several convolutional layers and fully connected layers, then softmax cross entropy, optimize using Adam with a learning rate of 1e-4). ...
4
votes
0answers
369 views

Dropout causes overfitting

I am trying to experiment with dropout in 2 layer NN on notMNIST dataset using TensorFlow (assignment 3 in Google Deep Learning Course on Udacity). But adding dropout causes fall in test accuracy and ...
3
votes
0answers
120 views

Why do Srivastava et al. claim that “the best” theoretical regularization technique involves all possible network parameter settings?

In the original paper on Dropout by Srivastava, Hinton, Krizhevsky et al. (2014), the authors make this claim in the introduction: With unlimited computation, the best way to "regularize" a fixed-...
11
votes
2answers
5k views

How is Spatial Dropout in 2D implemented?

This is with refernce to the paper Efficient Object Localization Using Convolutional Networks, and from what I understand the dropout is implemented in 2D. After reading the code from Keras on how ...
3
votes
2answers
408 views

Dropout effectiveness on small neural networks

I implemented dropout on my neural networks. I tried to train the neural network to act as the f(x) = sin(x) function. During normal backpropagation without dropout ...
2
votes
2answers
885 views

If I use dropout in a neural network and run it for a large number of steps, do I risk deleting all the units?

I'm still trying to understand dropout completely, but this is what think is happening so far: At each step there is a chance p of a unit being set to zero. If a rectified linear unit (ReLU) is used ...