Skip to main content

Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network. DO NOT use this tag for dropout as in censoring or missing data in survival analysis or longitudinal data analysis.

Filter by
Sorted by
Tagged with
-1 votes
1 answer
23 views

How does a neural network differentiate between a neuron that outputs 0 and a dropped-out one?

How does a network differentiate between a neuron with output 0 and a dropped-out neuron (this neuron might output a non-zero value but due to dropout it outputs 0)?
ado sar's user avatar
  • 477
0 votes
1 answer
54 views

Drop-outs in RCT

I have conducted a RCT with 34 participants, experimental group and a control group. We took measurement data before randomisation at baseline (T0), and then follow-up measurements every 4 months (so ...
Mai's user avatar
  • 21
0 votes
0 answers
19 views

time series cross validation and avoidance of overfitting

So. I am doing Time Series Classification on various datasets using different types of classifiers (deep learning, dictionary-based, distance-based, interval-based, feature-based, convolution). As far ...
Sophia Vei's user avatar
1 vote
0 answers
36 views

Reproducing results from classic dropout paper [closed]

In the classic paper "Dropout: A Simple Way to Prevent Neural Networks from Overfitting", there is a figure comparing the features learned by a one-layer autoencoder trained on MNIST with ...
Ari Herman's user avatar
0 votes
0 answers
20 views

MC Dropout without weight decay -- how is the model precision calculated?

In the original [MC Dropout paper][1], the variance is calculated as the sum of two contributions, an sample variance (over the multiple forward passes), var$(y)$, plus a term quantifying the inverse ...
till-m's user avatar
  • 1
1 vote
1 answer
354 views

Are dropout layers equivalent to adding noise to training samples?

I'm doing reading around regularisation techniques for neural networks. My intuition is that dropout is essentially adding noise into the network by zeroing out activations according to a given ...
Molem7b5's user avatar
0 votes
1 answer
57 views

Does the phrase "Training Example" always mean a Single Row?

I'm trying to understand the Dropout algorithm. In this paper, the authors say that the nodes are randomly switched off with probability $p$ for each "Training Example". Does this literally ...
Connor's user avatar
  • 655
0 votes
0 answers
160 views

How is Dropout implemented in the Training Algorithm? [duplicate]

I've searched for an answer to this, and read a number of scientific articles on the subject, but I can't find a practical explanation of how Dropout actually drops nodes in an algorithm. I've read ...
Connor's user avatar
  • 655
2 votes
1 answer
482 views

Uncertainty score from Monte Carlo dropout

When using a neural network for multi-class classification, there are situations where it is useful to estimate the uncertainty of the network's predicted class. One leading method for estimating ...
D.W.'s user avatar
  • 6,688
6 votes
3 answers
2k views

If dropout is going to remove neurons, why are those neurons built?

I know that Dropout will remove neurons randomly to reduce over-fitting. If Dropout is going to remove neurons, why are those neurons built? We could remove those neurons from the architecture. Why ...
Naren Babu R's user avatar
1 vote
1 answer
23 views

More dense layers with heavy dropouts or fewer layers with light dropouts?

I'm trying to build a network. While creating the fully connected part in the last, Which one should we prefer: More layers that regularly reduce with heavy dropouts or fewer layers that reduce ...
Salim Azak's user avatar
2 votes
0 answers
67 views

Can recurrent dropout be used as a Bayesian approximation?

To express model uncertainty in LSTM, we can use dropout as a Bayesian approximation for Gaussian model according to this, (dropout is kept during inference and that will result in a distribution of ...
Traveling Salesman's user avatar
1 vote
0 answers
860 views

Where to add Dropout in CNN-LSTM?

I am creating a CNN-LSTM model to forecast sequential simulation data. At the moment I am not sure what the best place is to use Dropout in a CNN-LSTM architecture. Is it between the CNN and LSTM ...
Quinten's user avatar
  • 389
3 votes
2 answers
4k views

Should I be using batchnorm and/or dropout in a VAE or GAN?

I am trying to design some generative NN models on datasets of RGB images and was debating on whether I should be using dropout and/or batch norm. Here are my thoughts (I may be completely wrong): ...
Aditya Mehrotra's user avatar
3 votes
2 answers
2k views

Does dropout have any benefits when overfitting isn't a concern?

I'm training a transformer based deep learning model in a regime where overfitting isn't a concern. Infinite training samples are generated on demand and never repeated, so there is no training ...
Sam's user avatar
  • 131
1 vote
1 answer
261 views

Training MLP by early-stopping without dropout layers

I am training a multi-layer perceptron (MLP) with 4 hidden layers. I got the best hyper-parameters by the following steps using HParams: Training model by each ...
Belter's user avatar
  • 185
3 votes
0 answers
183 views

Intuition behind random strides in CNNs

I recently attended a lecture on CNNs and was given a brief overview on the topic of dropout. I understood the logic behind the regularization and silencing the firing of neurons to prevent ...
Harduin's user avatar
  • 77
1 vote
1 answer
284 views

Dropout value - increase, decrease, keep the same across layers

I'm confused about dropout values that people set. Sometimes it's the same value, say 0.4. Sometimes they increase them gradually from 0.2 to 0.5. For example after ...
cytcytcy's user avatar
3 votes
2 answers
395 views

Why use dropout in feedforward?

Maybe I am just confused by what is the point of using dropout in the feed-forward? Wouldn't be better to forward the input with the whole network and then use the dropout only in the back-prop to ...
Alessandro Mondin's user avatar
2 votes
0 answers
793 views

In attention models, why is dropout used for positional encodings?

In the "Attention is all you need" paper, they write: we apply dropout to the sums of the embeddings and the positional encodings I can understand why you might use dropout on the ...
user1834069's user avatar
1 vote
1 answer
71 views

Approximating the posterior and learning the distribution over the weights after training

I am familiar with the methods in variational inference in which after training we have access to the distribution over the network's weights. This is necessary for estimating epistemic uncertainty. ...
samsambakster's user avatar
6 votes
1 answer
13k views

Where is dropout placed in the original transformer?

I wanted to know where dropout was placed in the original transformer. According to the original paper Attention Is All You Need they say: ...
Charlie Parker's user avatar
3 votes
1 answer
1k views

What can I do when Overfitting doesn't seem to go away by any means?

So first of all I've seen a lot of overfitting questions around here, but none of the answers seem to improve my model. I wrote a neural network made without frameworks (only used numpy), and for the ...
Ramiro's user avatar
  • 93
0 votes
0 answers
132 views

Can I tell my model is overfittng?

I am developing fully convolutional model for semantic segmentation task and I tried to use spatial dropout layers to prevent overfitting of my model. My model has interesting learning curves and I am ...
Many's user avatar
  • 113
0 votes
1 answer
632 views

Quality of model and dropout

I am training my neural network (modified VGG) for solving the problem of 3-class classification. For reaching the best model I am trying to fit slightly different models. I have some questions about ...
AlexM's user avatar
  • 91
0 votes
0 answers
46 views

Regularising neural networks with pre-trained weights

I trained CNN to over-fit the data (close to 0 train error), it took 6 days to train. Now I want to regularise the model using L2 or Dropout, with weights initialised from over-fitted model, will it ...
spb's user avatar
  • 11
5 votes
1 answer
7k views

Using batchnorm and dropout simultaneously?

I am a bit confused about the relation between terms "Dropout" and "BatchNorm". As I understand, Dropout is regularization technique, which is using only during training. ...
AlexM's user avatar
  • 91
0 votes
0 answers
44 views

Training accuracy improves but validation and test accuracy don't

I am training an LSTM classifier on a binary data set in order to predict if a body-headline set is related or unrelated. The headline and body were combined as a string and then the features were ...
Alex K's user avatar
  • 11
0 votes
0 answers
60 views

Network topology of dropout

So dropout is a popular way to regularize neural networks by randomly removing nodes in the network. There are similar methods that remove edges, as well as skip connections which introduce ...
900edges's user avatar
  • 399
1 vote
0 answers
37 views

Can dropout negatively impact performance by increasing repetition?

Dropout is the idea that you can drop, i.e set to zero, some of the nodes in a computational neural network. The goal of this is to increase regularization by preventing the model from relying too ...
Joshiepillow's user avatar
4 votes
1 answer
2k views

Attention dropout, where was it proposed/used first?

Attention dropout (dropout on the attention weights) is very common for the Transformer model. In the original Attention is all you need paper, dropout is mentioned, but not for the attention weights. ...
Albert's user avatar
  • 1,255
1 vote
1 answer
360 views

Dropout in highly unbalanced longitudinal data (WGEE)

I have found a lot of software and examples that uses Weighted Generalized Estimating Equations to deal with missing data in a balanced data set (equal time points). However, I have a very high ...
MDG1999's user avatar
  • 105
1 vote
0 answers
422 views

Order of dropout and activation in 1D convolutional networks

I have a simple cnn-lstm network. There are two 1D convolutional layers after the input layer. Every 1D convolutional layer is followed by a dropout. What I observe is that when I have conv1D -> ...
Miranda's user avatar
  • 176
1 vote
0 answers
73 views

Convolutional NN binary classifier makes predictions between 0.3 and 1, expected 0-1

I'm training a multiple input model for binary classification using Conv2D for one of the branches, and the results are very good on the validation set. I am using Dropout of 0.5 just before the ...
Mgnum33's user avatar
  • 11
0 votes
0 answers
40 views

Is it possible to resize/add noise to a hidden layer during training?

I'm working on class for a project where my basic idea is to create a "drop-in" layer, so similar to drop out, I'm thinking of adding hidden units that are sampled from the same weight ...
skidjoe's user avatar
  • 211
4 votes
1 answer
235 views

Does Dropout need a validation set to prevent overfitting? [closed]

Is it really necessary to use a validation set to avoide overfitting while we are using Dropout ?
Ka_'s user avatar
  • 165
1 vote
0 answers
93 views

How to re-scale the inputs for forward-propagation and backpropagation in the drop out?

Assume $p$ is the keep probability for drop out, for the forward-propagation, we do the scaling for the inputs as $A_r = A/p$. In the backpropagation, as many other people said (dropout: forward prop ...
Harry's user avatar
  • 227
2 votes
1 answer
146 views

MC dropout deep learning

I would like to compute the uncertainty of my deep learning model using MC dropout. My original model contains already one dropout and I am satisfied with its performance. To compute the uncertainty, ...
axel's user avatar
  • 21
1 vote
1 answer
130 views

What is standard dropout with weight averaging?

I understand that standard dropout is switching neurons on and off during training, to minimize overfitting. I came across standard dropout with weight averaging in this paper (Figure 2a), what is ...
MJimitater's user avatar
4 votes
1 answer
165 views

Is dropout (deep learning) even consistent?

I'm learning about dropout from these sources: https://cs231n.github.io/neural-networks-2/#reg https://arxiv.org/abs/1207.0580 At test time, the trained weights are scaled by a factor of $p$, to ...
user357269's user avatar
3 votes
1 answer
570 views

why does drop-out increase training time?

The textbook I am reading just says that dropout tends to increase the total training time by 2~3 times but does not explain why. Can someone provide an intuitive explanation?
DSKim's user avatar
  • 1,289
4 votes
1 answer
4k views

Should the Dropout vs. Recurrent Dropout Arguments Be the Same in Keras?

I'm learning about recurrent neural networks right now, and am in chapter 6 of Deep Learning with Python by Francois Chollet. In the chapter it's discussing using dropout in recurrent layers. I ...
Jonathan Bechtel's user avatar
1 vote
0 answers
23 views

zero amputation interpretation as dropout

Assume I'm trying to train a neural net on training data with some missing features across the training samples. Assume that I replace the missing feature with zero. Is this equal to the usage of ...
Dannynis's user avatar
1 vote
0 answers
413 views

Quantify the output variance of a neural network classifier

Lately at work we are dealing with a theoretical problem concerning the output variance of a neural network classifier. To set the scene, suppose you have an image classifier, which takes an image as ...
CedricDeBoom's user avatar
4 votes
1 answer
981 views

What is the Cost Function for Neural Network with Dropout Regularisation?

For some context, I shall outline my current understanding: Considering a Neural Network, for a Binary Classification problem, the Cross-entropy cost function, J, is defined as: $ J = \frac{-1}{m} \...
Nitin's user avatar
  • 349
1 vote
1 answer
3k views

Does ReLU produce the same effect as dropouts?

When we add dropouts to a densely connect layer, it randomly ignores nodes, by considering their output to always be zero. Though we may not observe the exact same effect in a CNN with ReLU as its ...
Sreram's user avatar
  • 135
3 votes
2 answers
753 views

Computation time with respect to Dropout

I've been recently attempting to speed up neural network training (in PyTorch). My question is the following. Does the computation time of a given feedforward neural network vary based on Dropout ...
sdgaw erzswer's user avatar
0 votes
0 answers
22 views

Is this expression for Loss valid?

The negative likelihood loss over training set $t$ where a training instance is given by $x^{(t)}$, taget by $y^{(t)}$, a specific masking (dropout) by $m$ and weights by $w$ as: $$L(w|m) = -\sum_t ...
DuttaA's user avatar
  • 215
6 votes
0 answers
839 views

Usage of dropout in convolutional GANs with batch norm?

In DCGAN, dropout is not used in either generator or discriminator. When using batch norm, are the benefits of dropout generally so marginal that is is not used? If it is used, in what circumstances?...
Tom Hale's user avatar
  • 2,611
5 votes
4 answers
611 views

Creating thinned models in during Dropout process

Applying dropout to a neural network amounts to sampling a “thinned” network from it. The thinned network consists of all the units that survived dropout. A neural net with $n$ units can be seen as a ...
ashirwad's user avatar