Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network. DO NOT use this tag for dropout as in censoring or missing data in survival analysis or longitudinal data analysis.

Filter by
Sorted by
Tagged with
1 vote
0 answers
56 views

Where to add Dropout in CNN-LSTM?

I am creating a CNN-LSTM model to forecast sequential simulation data. At the moment I am not sure what the best place is to use Dropout in a CNN-LSTM architecture. Is it between the CNN and LSTM ...
user avatar
  • 121
2 votes
1 answer
85 views

Should I be using batchnorm and/or dropout in a VAE or GAN?

I am trying to design some generative NN models on datasets of RGB images and was debating on whether I should be using dropout and/or batch norm. Here are my thoughts (I may be completely wrong): ...
user avatar
2 votes
2 answers
209 views

Does dropout have any benefits when overfitting isn't a concern?

I'm training a transformer based deep learning model in a regime where overfitting isn't a concern. Infinite training samples are generated on demand and never repeated, so there is no training ...
user avatar
  • 121
1 vote
1 answer
61 views

Training MLP by early-stopping without dropout layers

I am training a multi-layer perceptron (MLP) with 4 hidden layers. I got the best hyper-parameters by the following steps using HParams: Training model by each ...
user avatar
  • 135
3 votes
0 answers
136 views

Intuition behind random strides in CNNs

I recently attended a lecture on CNNs and was given a brief overview on the topic of dropout. I understood the logic behind the regularization and silencing the firing of neurons to prevent ...
user avatar
  • 65
1 vote
1 answer
30 views

Dropout value - increase, decrease, keep the same across layers

I'm confused about dropout values that people set. Sometimes it's the same value, say 0.4. Sometimes they increase them gradually from 0.2 to 0.5. For example after ...
user avatar
0 votes
0 answers
110 views

Uncertainty Quantification in conditional VAE

I would like to collect some thoughts and references on how to quantify the uncertainties in predictions of neural network based models. In particular, I am using a conditional VAE to translate ...
user avatar
  • 123
2 votes
1 answer
102 views

Why use dropout in feedforward?

Maybe I am just confused by what is the point of using dropout in the feed-forward? Wouldn't be better to forward the input with the whole network and then use the dropout only in the back-prop to ...
user avatar
1 vote
0 answers
16 views

In attention models, why is dropout used for positional encodings?

In the "Attention is all you need" paper, they write: we apply dropout to the sums of the embeddings and the positional encodings I can understand why you might use dropout on the ...
user avatar
0 votes
0 answers
9 views

Help to make sense over increasing validation loss on linear model over BERT embeddings [duplicate]

I am trying to use BERT generated embeddings in a simple linear model with relu and dropouts(0.3) in between two hidden layers of dimensions 256 and 128, respectively. For a binary classification task....
user avatar
1 vote
1 answer
31 views

Approximating the posterior and learning the distribution over the weights after training

I am familiar with the methods in variational inference in which after training we have access to the distribution over the network's weights. This is necessary for estimating epistemic uncertainty. ...
user avatar
0 votes
0 answers
126 views

Could I apply Spatial or simple Dropout before or after Adaptive Average Pooling or Global Average Pooling?

I'm working on a 1D CNN and I want to apply a Monte Carlo Dropout in order to get the mean of the predictions for each instance (as well as the variance, and entropy later on). The network topology ...
user avatar
0 votes
0 answers
33 views

How to better regularize a 1D-CNN for Gesture Recognition Time Series Data?

I’m currently developing a 1D-CNN to work on a Gesture Recognition approach and I’m trying to solve the problem of how to regularize correctly the Neural Network and recognize the known unknowns while ...
user avatar
2 votes
1 answer
2k views

Where is dropout placed in the original transformer?

I wanted to know where dropout was placed in the original transformer. According to the original paper (https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) they say: <...
user avatar
3 votes
1 answer
132 views

What can I do when Overfitting doesn't seem to go away by any means?

So first of all I've seen a lot of overfitting questions around here, but none of the answers seem to improve my model. I wrote a neural network made without frameworks (only used numpy), and for the ...
user avatar
0 votes
0 answers
58 views

Can I tell my model is overfittng?

I am developing fully convolutional model for semantic segmentation task and I tried to use spatial dropout layers to prevent overfitting of my model. My model has interesting learning curves and I am ...
user avatar
  • 113
0 votes
1 answer
68 views

Quality of model and dropout

I am training my neural network (modified VGG) for solving the problem of 3-class classification. For reaching the best model I am trying to fit slightly different models. I have some questions about ...
user avatar
  • 61
0 votes
0 answers
15 views

Regularising neural networks with pre-trained weights

I trained CNN to over-fit the data (close to 0 train error), it took 6 days to train. Now I want to regularise the model using L2 or Dropout, with weights initialised from over-fitted model, will it ...
user avatar
  • 11
2 votes
1 answer
703 views

Using batchnorm and dropout simultaneously?

I am a bit confused about the relation between terms "Dropout" and "BatchNorm". As I understand, Dropout is regularization technique, which is using only during training. ...
user avatar
  • 61
0 votes
0 answers
32 views

Training accuracy improves but validation and test accuracy don't

I am training an LSTM classifier on a binary data set in order to predict if a body-headline set is related or unrelated. The headline and body were combined as a string and then the features were ...
user avatar
  • 11
0 votes
0 answers
32 views

Network topology of dropout

So dropout is a popular way to regularize neural networks by randomly removing nodes in the network. There are similar methods that remove edges, as well as skip connections which introduce ...
user avatar
  • 349
1 vote
0 answers
27 views

Can dropout negatively impact performance by increasing repetition?

Dropout is the idea that you can drop, i.e set to zero, some of the nodes in a computational neural network. The goal of this is to increase regularization by preventing the model from relying too ...
user avatar
2 votes
0 answers
121 views

Attention dropout, where was it proposed/used first?

Attention dropout (dropout on the attention weights) is very common for the Transformer model. In the original Attention is all you need paper, dropout is mentioned, but not for the attention weights. ...
user avatar
  • 1,155
0 votes
1 answer
135 views

Dropout in highly unbalanced longitudinal data (WGEE)

I have found a lot of software and examples that uses Weighted Generalized Estimating Equations to deal with missing data in a balanced data set (equal time points). However, I have a very high ...
user avatar
  • 95
1 vote
0 answers
149 views

Order of dropout and activation in 1D convolutional networks

I have a simple cnn-lstm network. There are two 1D convolutional layers after the input layer. Every 1D convolutional layer is followed by a dropout. What I observe is that when I have conv1D -> ...
user avatar
  • 156
1 vote
0 answers
54 views

Convolutional NN binary classifier makes predictions between 0.3 and 1, expected 0-1

I'm training a multiple input model for binary classification using Conv2D for one of the branches, and the results are very good on the validation set. I am using Dropout of 0.5 just before the ...
user avatar
  • 11
0 votes
0 answers
17 views

Is it possible to resize/add noise to a hidden layer during training?

I'm working on class for a project where my basic idea is to create a "drop-in" layer, so similar to drop out, I'm thinking of adding hidden units that are sampled from the same weight ...
user avatar
  • 181
4 votes
1 answer
122 views

Does Dropout need a validation set to prevent overfitting? [closed]

Is it really necessary to use a validation set to avoide overfitting while we are using Dropout ?
user avatar
  • 65
1 vote
0 answers
45 views

How to re-scale the inputs for forward-propagation and backpropagation in the drop out?

Assume $p$ is the keep probability for drop out, for the forward-propagation, we do the scaling for the inputs as $A_r = A/p$. In the backpropagation, as many other people said (dropout: forward prop ...
user avatar
  • 205
1 vote
1 answer
66 views

MC dropout deep learning

I would like to compute the uncertainty of my deep learning model using MC dropout. My original model contains already one dropout and I am satisfied with its performance. To compute the uncertainty, ...
user avatar
  • 11
0 votes
0 answers
50 views

What is standard dropout with weight averaging?

I understand that standard dropout is switching neurons on and off during training, to minimize overfitting. I came across standard dropout with weight averaging in this paper (Figure 2a), what is ...
user avatar
4 votes
1 answer
117 views

Is dropout (deep learning) even consistent?

I'm learning about dropout from these sources: https://cs231n.github.io/neural-networks-2/#reg https://arxiv.org/abs/1207.0580 At test time, the trained weights are scaled by a factor of $p$, to ...
user avatar
3 votes
1 answer
228 views

why does drop-out increase training time?

The textbook I am reading just says that dropout tends to increase the total training time by 2~3 times but does not explain why. Can someone provide an intuitive explanation?
user avatar
  • 1,139
2 votes
1 answer
3k views

Should the Dropout vs. Recurrent Dropout Arguments Be the Same in Keras?

I'm learning about recurrent neural networks right now, and am in chapter 6 of Deep Learning with Python by Francois Chollet. In the chapter it's discussing using dropout in recurrent layers. I ...
user avatar
1 vote
0 answers
18 views

zero amputation interpretation as dropout

Assume I'm trying to train a neural net on training data with some missing features across the training samples. Assume that I replace the missing feature with zero. Is this equal to the usage of ...
user avatar
1 vote
0 answers
288 views

Quantify the output variance of a neural network classifier

Lately at work we are dealing with a theoretical problem concerning the output variance of a neural network classifier. To set the scene, suppose you have an image classifier, which takes an image as ...
user avatar
2 votes
1 answer
509 views

What is the Cost Function for Neural Network with Dropout Regularisation?

For some context, I shall outline my current understanding: Considering a Neural Network, for a Binary Classification problem, the Cross-entropy cost function, J, is defined as: $ J = \frac{-1}{m} \...
user avatar
  • 159
0 votes
1 answer
1k views

Does ReLU produce the same effect as dropouts?

When we add dropouts to a densely connect layer, it randomly ignores nodes, by considering their output to always be zero. Though we may not observe the exact same effect in a CNN with ReLU as its ...
user avatar
  • 125
3 votes
2 answers
452 views

Computation time with respect to Dropout

I've been recently attempting to speed up neural network training (in PyTorch). My question is the following. Does the computation time of a given feedforward neural network vary based on Dropout ...
user avatar
0 votes
0 answers
20 views

Is this expression for Loss valid?

The negative likelihood loss over training set $t$ where a training instance is given by $x^{(t)}$, taget by $y^{(t)}$, a specific masking (dropout) by $m$ and weights by $w$ as: $$L(w|m) = -\sum_t ...
user avatar
  • 215
6 votes
0 answers
636 views

Usage of dropout in convolutional GANs with batch norm?

In DCGAN, dropout is not used in either generator or discriminator. When using batch norm, are the benefits of dropout generally so marginal that is is not used? If it is used, in what circumstances?...
user avatar
  • 2,271
5 votes
4 answers
360 views

Creating thinned models in during Dropout process

Applying dropout to a neural network amounts to sampling a “thinned” network from it. The thinned network consists of all the units that survived dropout. A neural net with $n$ units can be seen as a ...
user avatar
10 votes
1 answer
2k views

Dropout in Linear Regression

I've been reading the original paper on dropout, (https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) and in the linear regression section, it is stated that: $\mathbb{E}_{R\sim Bernoulli(p)}\...
user avatar
  • 199
0 votes
2 answers
326 views

How many neurons are actually dropped when using dropout?

I understand that when using dropout, a single neuron can be described using Bernoulli random variable and for a set of neurons it can be described as Binomial random variable When using Dropout, we ...
user avatar
1 vote
1 answer
135 views

Theoretical foundation for dropout in neural networks

Can someone point me to a thorough theoretical foundation for dropout in training neural networks? So far I have found only handwaving explanations (e.g. Goodfellow's textbook and the original paper) ...
user avatar
4 votes
1 answer
2k views

Where to include Dropout in stacked autoencoder

I'm using Keras to implement a stacked autoencoder, and I think it may be overfitting. I wanted to include dropout, and keep reading about the use of dropout in autoencoders, but I cannot find any ...
user avatar
  • 85
2 votes
1 answer
384 views

What does it mean by "approach the performance of the Bayesian gold standard"?

It is a sentence in Dropout paper(Dropout: A Simple Way to Prevent Neural Networks from Overfitting). "This can sometimes be approximated quite well for simple or small models, but we would like to ...
user avatar
1 vote
0 answers
36 views

To remove neural-network units or to increase drop-out?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
user avatar
  • 922
1 vote
0 answers
206 views

How to apply dropout in LSTMs?

Dropout in fully connected neural networks is simpl to visualize, by just 'dropping' connections between units with some probability set by hyperparamter p. However, how dropout works in recurrent ...
user avatar
  • 922
3 votes
0 answers
218 views

Dropout: Redundant representation vs. breaking up co-adaptation

Dropout is commonly used to regularize NNs. On one hand Dropout forces the network to have a redundant representation, since due to the dropout the NN can not rely on specific neurons for the ...
user avatar
  • 158