Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

# Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

358 questions
Filter by
Sorted by
Tagged with
248 views

### Simple Neural Network Issue in Python? [closed]

I thought it might be a good exercise to try my hand at making a simple, one hidden layer neural net from scratch. But, for whatever reason, I can't get my in-sample error to go down. I think it has ...
89 views

### Implementing backpropagation for dataset preprocessing

I have this algorithm written that can perform the XOR operation. How can i modify it so that it can process the large dataset? ...
530 views

### Scale MNIST-Data to [-0.9, 0.9]

I'm programming a neural network for MNIST-Recognition. My net has a pretty good performance, with accuracy > 98% on test set. But the training is very slow. So I thought it would be faster if I scale ...
788 views

### Fast RTRL(Real Time Recurrent Learning) for RNN

Assume generic RNN has update formula: $\mathbf{h}_{t+1} = f(\mathbf{x_t},\mathbf{h_t},\mathbf{\theta})$  Where $\mathbf{x}$ is input vector, $\mathbf{h}$ is hidden state vector, and $\theta$ is ...
395 views

### Does Adding more neural units reduce the probability of trapping in a local minima?

Consider a multi-layer neural network that learns its weights with backpropagation (and gradient descent). Hence, there is a probability that we trap into a local minimum. Will adding more neural ...
107 views

### Training a MLP after pretraining RBMs with dropout

Let's say I have a couple of RBMs that I pretrained and that I used dropout. When finetuning, how does having used dropout effect backpropagation? Do I still use dropout while backprogating and change ...
2k views

### resilient backpropagation parameters selection

In the original paper for resilient backpropagation (http://paginas.fe.up.pt/~ee02162/dissertacao/RPROP%20paper.pdf), the author says "One of the main advantages of RPROP lies in the fact, that for ...
373 views

### Why is the second derivative required for newton's method for back-propagation?

I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely. I have seen this same question, and the widely accepted answer ...
256 views

I am in the process of implementing back propagation into my image classification neural net. I am using this cost function with a sigmoid output layer and ReLU hidden layers. The neural net has 3 ...
4k views

### Dropout backpropagation implementation

I understood the feedforward part of dropout during training, where for each example I multiply each activation with a binary mask to de-activate neurons with probability p. I use the inverted ...
234 views

### Derivative of the loss function w.r.t to X for the backpropagation

I would like to ask you why do we need to calculate a derivative of the loss function w.r.t X? It seems like, that for the backpropagation we need to calculate only a derivative w.r.t W. Can you ...
440 views

### How does backpropagation work in the case of reinforcement learning for games?

If we want a neural network to learn how to recognize e.g. digits, the backpropagation procedure is as follows: Let the NN look at an image of a digit, and output its probabilities on the different ...
148 views

### In Nielsen's explanation of backpropagation, why does the way he defines error change? Is it actually a change?

Specifically, why does Equation (BP1) not have the same form as (29)?1 In order to explain backpropagating the gradient through the neural net, he starts by defining something he calls error (...
733 views

### Implementing backpropagation in Theano

I am new to the machine learning field, so I am not sure if I am asking a dumb question. I have been playing around with Theano for a while and read a lot of code examples and it looks like every time ...
70 views

### What does Goodfellow mean by “generator conditional variance”?

In Goodfellow's Generative Adversarial Nets, it is mentioned that Our work backpropagates derivatives through generative processes by using the observation that \lim_{\sigma \rightarrow 0} \...
45 views

### Validate implementation of back-propagation algorithm

Let's say I implemented a CNN. Is there an easy way I can validate, that my implementation of back-propagation does not contain errors ? May be I can feed some dummy values into my network so it can ...
195 views

219 views

### Why don't neural networks get stuck in loops when they overshoot a backprop step?

In a normal feedforward network I wrote with linear activations I've noticed that after a while when the network has found a pretty viable solution to a problem it sometimes takes a step in the wrong ...
61 views

### The nature of the problem of vanishing gradients in RNN

In the context of RNNs, gradient vanishing refers to the fact the gradient signal decays to zero as we approach the beginning of the sequence during the unfolding of the network in backpropagation ...
629 views

### How does backpropagation differ from reverse-mode autodiff

Going through this book, I am familiar with the following: For each training instance the backpropagation algorithm first makes a prediction (forward pass), measures the error, then goes through ...
345 views

### Backpropagation in multi-layer perceptron (MLP) doesn't converge [closed]

My simple fully-connected multilayer perceptron (MLP) that I'm writing for academic purposes is causing to me sleep deprivation. I can't figure out why my MLP learns poorly, even if I try to solve ...
1k views

### Neural Network - Success after changing weights initialization strategy. What's the explanation?

I'm implementing a neural network in javascript to recognize handwritten digits, while studying "Neural Networks and Deep Learning" by Michael Nielsen and following the feedforward-backpropagation ...
428 views

### Explain this backpropagation graph?

Can someone help explain this backpropagation graph? This is from cs231n Convolutional Neural Networks for Visual Recognition by Stanford. So for this graph, let's say the true value of the output ...
308 views

### FeedForward Alternative to Backpropagation

I am reading a blog post that tries to explain backpropagation. In the build up the author shows how a naive method for computing gradients is sub-optimal. Consider this: Naive feedforward ...
226 views

### why should we transfer final state to initial state (BPTT) in LSTM?

I am learning LSTM implementation in torch from this code,it has these two lines of code: ...
247 views