Questions tagged [dropout]
Dropout is a technique to reduce overfitting during the training phase of a neural network. DO NOT use this tag for dropout as in censoring or missing data in survival analysis or longitudinal data analysis.
129
questions
0
votes
0
answers
15
views
MC Dropout without weight decay -- how is the model precision calculated?
In the original [MC Dropout paper][1], the variance is calculated as the sum of two contributions, an sample variance (over the multiple forward passes), var$(y)$, plus a term quantifying the inverse ...
- 1
0
votes
0
answers
20
views
In what conditions do MC-Dropout-based uncertainties fail to be expressive?
I trained a neural network with dropout regularization and computed uncertainty scores for predictions on a test set. However, I found that these uncertainties exhibit a very weak correlation (Pearson:...
- 799
0
votes
0
answers
11
views
Does generalization methods in ANN increase variance in results?
I am new to generalization techniques in machine learning and I have been experimenting with them recently. I have observed that when I apply them to my model, the performance of the model improves, ...
- 13
1
vote
1
answer
149
views
Are dropout layers equivalent to adding noise to training samples?
I'm doing reading around regularisation techniques for neural networks. My intuition is that dropout is essentially adding noise into the network by zeroing out activations according to a given ...
- 91
0
votes
0
answers
16
views
Does the application of Dropout during inference changes the shape of the predictive distribution?
Suppose i have a set of data points (x,y) and i model the likelihood of the dataset as the product of independent gaussian distributions with different mean and variances.
Suppose now that i have ...
- 325
0
votes
0
answers
27
views
Dropout equivalence of averaging ensemble networks
In the dropout paper by Hinton et al. 2012 it's stated that:
"In networks with a single hidden layer of N units and a “softmax”
output layer for computing the probabilities of the class labels,
...
- 3,271
0
votes
0
answers
53
views
Recurrent dropout in keras: is the Semeniuta method equivalent to the Gal and Ghahramani method from the point of view of variational inference?
as the title. from this post https://stackoverflow.com/questions/44924690/keras-the-difference-between-lstm-dropout-and-lstm-recurrent-dropout I have recently learned that the implementation of ...
- 325
0
votes
0
answers
12
views
a question on model comparison when 2 different training techniques are used (dropout and variational inference)
i have a doubt :
considering 2 different neural networks, one trained through the variational inference technique with denseflipout layers and the other through dropout/concrete dropout
In the first ...
- 325
0
votes
1
answer
39
views
Does the phrase "Training Example" always mean a Single Row?
I'm trying to understand the Dropout algorithm. In this paper, the authors say that the nodes are randomly switched off with probability $p$ for each "Training Example".
Does this literally ...
- 451
0
votes
0
answers
117
views
How is Dropout implemented in the Training Algorithm? [duplicate]
I've searched for an answer to this, and read a number of scientific articles on the subject, but I can't find a practical explanation of how Dropout actually drops nodes in an algorithm.
I've read ...
- 451
2
votes
1
answer
288
views
Uncertainty score from Monte Carlo dropout
When using a neural network for multi-class classification, there are situations where it is useful to estimate the uncertainty of the network's predicted class. One leading method for estimating ...
- 6,578
6
votes
3
answers
2k
views
If dropout is going to remove neurons, why are those neurons built?
I know that Dropout will remove neurons randomly to reduce over-fitting.
If Dropout is going to remove neurons, why are those neurons built? We could remove those neurons from the architecture.
Why ...
- 176
1
vote
1
answer
19
views
More dense layers with heavy dropouts or fewer layers with light dropouts?
I'm trying to build a network. While creating the fully connected part in the last, Which one should we prefer: More layers that regularly reduce with heavy dropouts or fewer layers that reduce ...
- 13
2
votes
0
answers
62
views
Can recurrent dropout be used as a Bayesian approximation?
To express model uncertainty in LSTM, we can use dropout as a Bayesian approximation for Gaussian model according to this, (dropout is kept during inference and that will result in a distribution of ...
1
vote
0
answers
756
views
Where to add Dropout in CNN-LSTM?
I am creating a CNN-LSTM model to forecast sequential simulation data. At the moment I am not sure what the best place is to use Dropout in a CNN-LSTM architecture. Is it between the CNN and LSTM ...
- 359
3
votes
2
answers
2k
views
Should I be using batchnorm and/or dropout in a VAE or GAN?
I am trying to design some generative NN models on datasets of RGB images and was debating on whether I should be using dropout and/or batch norm.
Here are my thoughts (I may be completely wrong):
...
- 207
3
votes
2
answers
1k
views
Does dropout have any benefits when overfitting isn't a concern?
I'm training a transformer based deep learning model in a regime where overfitting isn't a concern. Infinite training samples are generated on demand and never repeated, so there is no training ...
- 131
1
vote
1
answer
224
views
Training MLP by early-stopping without dropout layers
I am training a multi-layer perceptron (MLP) with 4 hidden layers. I got the best hyper-parameters by the following steps using HParams:
Training model by each ...
- 165
3
votes
0
answers
180
views
Intuition behind random strides in CNNs
I recently attended a lecture on CNNs and was given a brief overview on the topic of dropout. I understood the logic behind the regularization and silencing the firing of neurons to prevent ...
- 75
1
vote
1
answer
205
views
Dropout value - increase, decrease, keep the same across layers
I'm confused about dropout values that people set.
Sometimes it's the same value, say 0.4.
Sometimes they increase them gradually from 0.2 to 0.5. For example after ...
- 23
3
votes
2
answers
319
views
Why use dropout in feedforward?
Maybe I am just confused by what is the point of using dropout in the feed-forward? Wouldn't be better to forward the input with the whole network and then use the dropout only in the back-prop to ...
1
vote
0
answers
612
views
In attention models, why is dropout used for positional encodings?
In the "Attention is all you need" paper, they write:
we apply dropout to the sums of the embeddings and the positional encodings
I can understand why you might use dropout on the ...
- 613
1
vote
1
answer
56
views
Approximating the posterior and learning the distribution over the weights after training
I am familiar with the methods in variational inference in which after training we have access to the distribution over the network's weights. This is necessary for estimating epistemic uncertainty. ...
- 177
4
votes
1
answer
10k
views
Where is dropout placed in the original transformer?
I wanted to know where dropout was placed in the original transformer. According to the original paper Attention Is All You Need they say:
...
- 6,616
3
votes
1
answer
721
views
What can I do when Overfitting doesn't seem to go away by any means?
So first of all I've seen a lot of overfitting questions around here, but none of the answers seem to improve my model.
I wrote a neural network made without frameworks (only used numpy), and for the ...
- 83
0
votes
0
answers
126
views
Can I tell my model is overfittng?
I am developing fully convolutional model for semantic segmentation task and I tried to use spatial dropout layers to prevent overfitting of my model.
My model has interesting learning curves and I am ...
- 113
0
votes
1
answer
495
views
Quality of model and dropout
I am training my neural network (modified VGG) for solving the problem of 3-class classification. For reaching the best model I am trying to fit slightly different models.
I have some questions about ...
- 81
0
votes
0
answers
43
views
Regularising neural networks with pre-trained weights
I trained CNN to over-fit the data (close to 0 train error), it took 6 days to train. Now I want to regularise the model using L2 or Dropout, with weights initialised from over-fitted model, will it ...
- 11
4
votes
1
answer
5k
views
Using batchnorm and dropout simultaneously?
I am a bit confused about the relation between terms "Dropout" and "BatchNorm". As I understand,
Dropout is regularization technique, which is using only during training.
...
- 81
0
votes
0
answers
44
views
Training accuracy improves but validation and test accuracy don't
I am training an LSTM classifier on a binary data set in order to predict if a body-headline set is related or unrelated. The headline and body were combined as a string and then the features were ...
- 11
0
votes
0
answers
58
views
Network topology of dropout
So dropout is a popular way to regularize neural networks by randomly removing nodes in the network. There are similar methods that remove edges, as well as skip connections which introduce ...
- 399
1
vote
0
answers
36
views
Can dropout negatively impact performance by increasing repetition?
Dropout is the idea that you can drop, i.e set to zero, some of the nodes in a computational neural network. The goal of this is to increase regularization by preventing the model from relying too ...
- 119
2
votes
0
answers
1k
views
Attention dropout, where was it proposed/used first?
Attention dropout (dropout on the attention weights) is very common for the Transformer model. In the original Attention is all you need paper, dropout is mentioned, but not for the attention weights. ...
- 1,205
1
vote
1
answer
316
views
Dropout in highly unbalanced longitudinal data (WGEE)
I have found a lot of software and examples that uses Weighted Generalized Estimating Equations to deal with missing data in a balanced data set (equal time points). However, I have a very high ...
- 105
1
vote
0
answers
408
views
Order of dropout and activation in 1D convolutional networks
I have a simple cnn-lstm network. There are two 1D convolutional layers after the input layer. Every 1D convolutional layer is followed by a dropout. What I observe is that when I have conv1D -> ...
- 176
1
vote
0
answers
67
views
Convolutional NN binary classifier makes predictions between 0.3 and 1, expected 0-1
I'm training a multiple input model for binary classification using Conv2D for one of the branches, and the results are very good on the validation set. I am using Dropout of 0.5 just before the ...
- 11
0
votes
0
answers
36
views
Is it possible to resize/add noise to a hidden layer during training?
I'm working on class for a project where my basic idea is to create a "drop-in" layer, so similar to drop out, I'm thinking of adding hidden units that are sampled from the same weight ...
- 211
4
votes
1
answer
215
views
Does Dropout need a validation set to prevent overfitting? [closed]
Is it really necessary to use a validation set to avoide overfitting while we are using Dropout ?
- 65
1
vote
0
answers
90
views
How to re-scale the inputs for forward-propagation and backpropagation in the drop out?
Assume $p$ is the keep probability for drop out, for the forward-propagation, we do the scaling for the inputs as $A_r = A/p$.
In the backpropagation, as many other people said (dropout: forward prop ...
- 227
2
votes
1
answer
136
views
MC dropout deep learning
I would like to compute the uncertainty of my deep learning model using MC dropout. My original model contains already one dropout and I am satisfied with its performance. To compute the uncertainty, ...
- 21
1
vote
1
answer
115
views
What is standard dropout with weight averaging?
I understand that standard dropout is switching neurons on and off during training, to minimize overfitting.
I came across standard dropout with weight averaging in this paper (Figure 2a), what is ...
- 269
4
votes
1
answer
152
views
Is dropout (deep learning) even consistent?
I'm learning about dropout from these sources:
https://cs231n.github.io/neural-networks-2/#reg
https://arxiv.org/abs/1207.0580
At test time, the trained weights are scaled by a factor of $p$, to ...
- 413
3
votes
1
answer
514
views
why does drop-out increase training time?
The textbook I am reading just says that dropout tends to increase the total training time by 2~3 times but does not explain why. Can someone provide an intuitive explanation?
- 1,269
2
votes
1
answer
3k
views
Should the Dropout vs. Recurrent Dropout Arguments Be the Same in Keras?
I'm learning about recurrent neural networks right now, and am in chapter 6 of Deep Learning with Python by Francois Chollet.
In the chapter it's discussing using dropout in recurrent layers. I ...
- 297
1
vote
0
answers
23
views
zero amputation interpretation as dropout
Assume I'm trying to train a neural net on training data with some missing features across the training samples. Assume that I replace the missing feature with zero. Is this equal to the usage of ...
- 11
1
vote
0
answers
398
views
Quantify the output variance of a neural network classifier
Lately at work we are dealing with a theoretical problem concerning the output variance of a neural network classifier. To set the scene, suppose you have an image classifier, which takes an image as ...
- 11
4
votes
1
answer
901
views
What is the Cost Function for Neural Network with Dropout Regularisation?
For some context, I shall outline my current understanding:
Considering a Neural Network, for a Binary Classification problem, the Cross-entropy cost function, J, is defined as:
$ J = \frac{-1}{m} \...
- 349
0
votes
1
answer
2k
views
Does ReLU produce the same effect as dropouts?
When we add dropouts to a densely connect layer, it randomly ignores nodes, by considering their output to always be zero.
Though we may not observe the exact same effect in a CNN with ReLU as its ...
- 125
3
votes
2
answers
696
views
Computation time with respect to Dropout
I've been recently attempting to speed up neural network training (in PyTorch). My question is the following.
Does the computation time of a given feedforward neural network vary based on Dropout ...
- 1,389
0
votes
0
answers
22
views
Is this expression for Loss valid?
The negative likelihood loss over training set $t$ where a training instance is given by $x^{(t)}$, taget by $y^{(t)}$, a specific masking (dropout) by $m$ and weights by $w$ as:
$$L(w|m) = -\sum_t ...
- 215