A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network. DO NOT use this tag for dropout as in censoring or missing data in survival analysis or longitudinal data analysis.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Can I tune dropout rate by bayesian optimization?

I am thinking about using drop out in my DL model. But, how to find the best drop out rate? Can I find it with bayesian optimization? I mean, drop out is a regularization technique, I would expect ...
1
vote
0answers
21 views

Quantify the output variance of a neural network classifier

Lately at work we are dealing with a theoretical problem concerning the output variance of a neural network classifier. To set the scene, suppose you have an image classifier, which takes an image as ...
1
vote
0answers
26 views

What is the Cost Function for Neural Network with Dropout Regularisation?

For some context, I shall outline my current understanding: Considering a Neural Network, for a Binary Classification problem, the Cross-entropy cost function, J, is defined as: $ J = \frac{-1}{m} \...
0
votes
1answer
36 views

Does ReLU produce the same effect as dropouts?

When we add dropouts to a densely connect layer, it randomly ignores nodes, by considering their output to always be zero. Though we may not observe the exact same effect in a CNN with ReLU as its ...
0
votes
0answers
120 views

Transfer learning when Dropout in source model

Anyone can you pls help me to understand that. If we have dropout layers in between the convolutional layers of source model. When we do transfer learning from source to target model, the convotional ...
3
votes
2answers
49 views

Computation time with respect to Dropout

I've been recently attempting to speed up neural network training (in PyTorch). My question is the following. Does the computation time of a given feedforward neural network vary based on Dropout ...
0
votes
0answers
48 views

Intuitive reasoning behind inverted dropout in neural networks

I'm going through the deeplearning.ai course on Coursera and am trying to understand the intuitive reasoning behind inverted dropout in neural networks. Based on the lecture, my understanding is as ...
0
votes
0answers
19 views

Is this expression for Loss valid?

The negative likelihood loss over training set $t$ where a training instance is given by $x^{(t)}$, taget by $y^{(t)}$, a specific masking (dropout) by $m$ and weights by $w$ as: $$L(w|m) = -\sum_t ...
0
votes
0answers
27 views

Using variable dropout in Keras

I need to implement a system with variable dropout factor in Keras with TensorFlow as backend. The dropout factor should change for each batch so that the the dropout factor varies from 0.0 to 0.20 at ...
5
votes
0answers
235 views

Usage of dropout in convolutional GANs with batch norm?

In DCGAN, dropout is not used in either generator or discriminator. When using batch norm, are the benefits of dropout generally so marginal that is is not used? If it is used, in what circumstances?...
4
votes
3answers
243 views

Creating thinned models in during Dropout process

Applying dropout to a neural network amounts to sampling a “thinned” network from it. The thinned network consists of all the units that survived dropout. A neural net with n units can be seen as a ...
9
votes
1answer
509 views

Dropout in Linear Regression

I've been reading the original paper on dropout, (https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) and in the linear regression section, it is stated that: $\mathbb{E}_{R\sim Bernoulli(p)}\...
0
votes
2answers
43 views

How many neurons are actually dropped when using dropout?

I understand that when using dropout, a single neuron can be described using Bernoulli random variable and for a set of neurons it can be described as Binomial random variable When using Dropout, we ...
0
votes
0answers
70 views

How dose dropout affect Weights and Bias?

I applied dropout in my network , and it worked , but i can't interpret dropout effects on weight and bias, to be more specific , i can't interpret why appling droput and not applying dropout have a ...
0
votes
1answer
73 views

Theoretical foundation for dropout in neural networks

Can someone point me to a thorough theoretical foundation for dropout in training neural networks? So far I have found only handwaving explanations (e.g. Goodfellow's textbook and the original paper) ...
2
votes
1answer
507 views

Where to include Dropout in stacked autoencoder

I'm using Keras to implement a stacked autoencoder, and I think it may be overfitting. I wanted to include dropout, and keep reading about the use of dropout in autoencoders, but I cannot find any ...
0
votes
0answers
11 views

What to do when the activation is non-linear when rescaling to compensate for dropout?

At 54:24 of this video, it says that once there is non-linearity in the activation function, the expectation is not exact. So would it post a problem for non-linear activation? Then how would you ...
1
vote
1answer
146 views

What does it mean by “approach the performance of the Bayesian gold standard”?

It is a sentence in Dropout paper(Dropout: A Simple Way to Prevent Neural Networks from Overfitting). "This can sometimes be approximated quite well for simple or small models, but we would like to ...
1
vote
0answers
25 views

To remove neural-network units or to increase drop-out?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
1
vote
0answers
118 views

How to apply dropout in LSTMs?

Dropout in fully connected neural networks is simpl to visualize, by just 'dropping' connections between units with some probability set by hyperparamter p. However, how dropout works in recurrent ...
2
votes
0answers
93 views

Dropout: Redundant representation vs. breaking up co-adaptation

Dropout is commonly used to regularize NNs. On one hand Dropout forces the network to have a redundant representation, since due to the dropout the NN can not rely on specific neurons for the ...
3
votes
1answer
1k views

Why does dropout increase the training time per epoch in a neural network?

I'm training an MLP neural network with one hidden layer and batch gradient descent using Keras/Tensorflow. Applying dropout to the input layer increased the training time per epoch by about 25 %, ...
1
vote
1answer
79 views

Price Forecasting Problem

I am working on a project for price movement forecasting and I am stuck with poor quality predictions. At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the ...
0
votes
1answer
37 views

School Dropout Prediction

I have a dataset composed by several features group by some factors (academic, personal, economic). I would like to predict the risk (high, medium, low) of dropout and its respective risk percentage. ...
0
votes
1answer
211 views

Dropout in Deep Neural Networks [duplicate]

I was reading a paper published on Dropout. What I find difficulty in understanding that, In the training phase, a unit is present with a probability p and not ...
1
vote
0answers
23 views

Training an ANN further once it reaches 100 % accuracy on training set

I have a very simply question: Does it make sense to further train an ANN once it reaches an accuracy of 100 % on the training data? I'm facing a binary classification problem and read this article ...
1
vote
0answers
102 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
4
votes
1answer
336 views

Drop-out as a multiplicative noise in deep neural networks

I am reading Ian Goodfellow's deep learning book, and I cannot understand the following lines: Another important aspect of dropout is that the noise is multiplicative. If the noise were additive ...
1
vote
1answer
885 views

Does it make sense to use a dropout layer in a neural network for a regression to predict an absolute Error?

I am working on a regression problem where I try to predict an Error with a NN with as little calculation steps as possible. Currently I have an input layer consisting of 21 Neurons and a Dense Output ...
1
vote
1answer
133 views

Why is dropout causing my network to overfit so badly? [closed]

I've been experimenting with various simple neural networks to test their performance. When I use the following architecture, I'm getting some very bad test error, which looks like overfitting. $$\...
2
votes
0answers
79 views

Why is dropout a good fit for ReLU units?

I have read this before that dropout interacts well with ReLU and recently going through SNN paper i came acoss this again. to quote: Dropout fits well to rectified linear units, since zero is in the ...
0
votes
1answer
86 views

Value of a dropout neuron with bias

I'm implementing dropout using the paper by Srivastava et al. In it, the authors suggest that when a neuron is dropped out, it is temporarily removed from the network completely: However, when ...
1
vote
1answer
4k views

Deep Learning : Using dropout in Autoencoders?

I am working with autoencoders and have few confusions, I am trying different autoencoders like: fully_connected autoencoder convolutional autoencoder denoising autoencoder I have two datasets, one ...
0
votes
1answer
34 views

Dropout: when do we eliminate neurons?

As you know at the training step according to the Dropout technique we eliminate every neuron with probability $p$. The question is do we eliminate neurons on every training instance (or batch) (or ...
0
votes
0answers
28 views

Back-propagation in presence of dropout [duplicate]

Intuitively dropout make sense to me but I don't understand how backpropagation works in presence of dropout. It looks like at each training step we backpropagate gradients to parameters in the ...
1
vote
0answers
256 views

Dropout: scaling the activation versus inverting the alpha dropout

I have the same question as in this post: Dropout: scaling the activation versus inverting the dropout but for alpha dropouts: I would like to know if I need to apply the scale factor of $p$ when ...
1
vote
1answer
71 views

Why do we need dropout in deep networks?

I have read some general statements about the usefulness of Dropout but the issue is still very vague to me. It is always said that it prohibits co-adaptability of neurons, but why it should be a bad ...
0
votes
1answer
49 views

Dropout and calculating values

I have been reading about dropout and understand what happens when training the network but I don't understand how it would work in calculating the network. Can someone please explain how the process ...
1
vote
0answers
98 views

Understanding dropout method: one bask per batch, or more?

I was talking with someone I know about the dropout method, and I realized we had different conceptions of how it worked. My impression was that there is one mask sampled per minibatch. His impression ...
1
vote
1answer
1k views

Weight scale inference when using dropout

Suppose I'm using dropout, and at test time I decide to do "weight scaling inference" (the method of predicting using the full network with weights multiplied by $p$, where $p$ is the probability of ...
0
votes
1answer
500 views

Fast dropout: How to compute the mean and variance of the approximating Gaussian?

In the Fast Dropout article by Wang and Manning, they talk about approximating the input to a hidden layer by a Gaussian with the same mean and variance as the inputs to the layer (see page 5). But ...
0
votes
1answer
50 views

Dropout for LSTMs

I've been just told that using Dropout For LSTMs is not considered the right thing these days. Is it true? If yes, what is recommended for overfitting prevention with LSTMs? Thanks!
3
votes
2answers
6k views

Dropout before Batch Normalization?

In the last course of the Deep Learning Specialization on Coursera from Andrew Ng, you can see that he uses the following sequence of layers on the output of an LSTM layer: Dropout -> BatchNorm -> ...
2
votes
2answers
469 views

Deep learning high dropout causes high model confidence scores

I am training an NLP classifier that maps input sentences to 1 of 50 categories. The model is a CNN language model, in which each input example is a 2d tensor of sentence length by word embedding ...
6
votes
2answers
4k views

Confused about Dropout implementations in Tensorflow

I have a network whose input size is 100 and output size 2. Only these layers. I applied a dropout with keep_prob of 0.8 and I tried to understand the outcome. As expected, the dropout mask has ...
1
vote
1answer
1k views

Value of the keep probability when calculating loss with dropout

I'm training a small neural network (2 hidden layers) to classify the mnist images, and want to apply dropout regularization before my output layer. My first question: is it worth applying dropout ...
2
votes
1answer
1k views

Does it make sense to use dropout in last layer of regression neural network? [closed]

I have a neural network that I constructed in keras that goes from a LSTM recurrent layer > dropout > flattened > dense layer of 1 unit. Does this make sense to ...
0
votes
1answer
94 views

High validation accuracy without scaling paramters when using dropout

I was training a CNN network on German traffic sign classifier data. The architecture was- 3 convolutional layers with intermediate max pooling concatenated outputs of layer 2 and layer 3 to feed to ...
3
votes
2answers
928 views

How is the dropout method different than Random Forests?

I've come across something called a dropout method that involves setting a threshold parameter $p$ and then for each predictor in your training set, generate a uniform random number. If that uniform ...
1
vote
1answer
160 views

Should I apply dropout if learning on huge dataset?

I am training an LSTM neural network for nlm on a big dataset: the model has about 100M learnable parameters and the dataset consists of about 2G characters. Therefore it seems that overfitting ...