# Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

341 questions
Filter by
Sorted by
Tagged with
1answer
62 views

### Backpropagation - partial derivatives

I am currently reading the Neural networks and deep learning book by Michael Nielsen. I have a question regarding the backpropagation chapter: Background: He explains the influence of a neuron on ...
1answer
1k views

### How do Variational Auto Encoders backprop past the sampling step

From my understanding of VAE's, there's a step during training in the middle where, after the encoder produces a mean and standard deviation, random samples are drawn from the given learned ...
0answers
28 views

### Back-propagation in presence of dropout [duplicate]

Intuitively dropout make sense to me but I don't understand how backpropagation works in presence of dropout. It looks like at each training step we backpropagate gradients to parameters in the ...
1answer
530 views

### How does backpropagation differ from reverse-mode autodiff

Going through this book, I am familiar with the following: For each training instance the backpropagation algorithm first makes a prediction (forward pass), measures the error, then goes through ...
0answers
45 views

### Can Regularization by achieved using Relative Sensitivity?

In a Mathematical Model we measure the sensitivity of the output with respect to the parameters and it is desirable that a small change in a parameter doesn't lead to wild fluctuations in the output ...
1answer
310 views

### How does backpropagation work in the case of reinforcement learning for games?

If we want a neural network to learn how to recognize e.g. digits, the backpropagation procedure is as follows: Let the NN look at an image of a digit, and output its probabilities on the different ...
1answer
377 views

### how to derive the gradient of batch normalization

I'm trying to figure out the gradient of batch norm wrt x for backprop, but I get stuck in what I will call 'the triangle of (gradient) death'. I present to you the triangle of death (in red), in the ...
1answer
680 views

### What is the difference among stochastic, batch and mini-batch learning styles?

So far as I know, we have the following scenario: stochastic: The error is calculated for each sample s. So, we can calculate the gradients for s. And we can update the weights of the network ...
0answers
798 views

### Numerical gradient checking (best practices)

I've implemented a neural network and am using numerical gradient checking to validate the back-propagation algorithm is working correctly. I'm using the standard method to calculate the numerical ...
1answer
94 views

### Intuitive Explanation for Gradients in ReLU-Activated Networks

When calculating gradients for backpropagation, a surprising result is that the gradient of the weights in a ReLU-activated layer ends up being a diagonal matrix. I am seeking to understand how this ...
4answers
16k views

### Why is tanh almost always better than sigmoid as an activation function?

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$. The reason he gives is that the outputs using $tanh$ ...
1answer
220 views

### Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
1answer
1k views

1answer
1k views

### Matrix form of backpropagation with batch normalization

Batch normalization has been credited with substantial performance improvements in deep neural nets. Plenty of material on the internet shows how to implement it on an activation-by-activation basis. ...
1answer
1k views

### Non zero centered activation functions

I read the following section from cs231n course notes: Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this ...
0answers
321 views

### finding the loss derivative w.r.t. weights for a convolutional layer

Take the loss function: $$\mathrm{loss} ~~=~~ \sum_{i=1}^N \left( -z_{}[y] + \log{\left( \sum_{c=1}^{10} \exp(z_{}[c]) \right)} \right)$$ where $z \in \mathbb{R}^{10}$ is the input to the softmax ...
1answer
20k views

### How is softmax_cross_entropy_with_logits different from softmax_cross_entropy_with_logits_v2?

Specifically, I suppose I wonder about this statement: Future major versions of TensorFlow will allow gradients to flow into the labels input on backprop by default. Which is shown when I ...
0answers
3k views

### Backpropagation on a convolutional layer

Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually). First let me try ...
1answer
84 views

### Neural network gives incorrect outputs although its backpropagates correctly

I am trying to program an OCR based on neural networks. It has in sum 3 layers : Input( 32 * 32 ) => Hidden (180) => Output(2) It task is to recognize the numbers 1 and 2. If I let the network train ...
1answer
77 views

### In Deep Learning, how much does randomness help generalization?

Let's assume at the beginning the NN is initialized with random weights, then the Backpropagation "shapes" the weights but the signal is strongest close to the final layer (where it gets computed) and ...
2answers
233 views

### Backpropagation gradients don't match approximated gradients

I am in the process of implementing back propagation into my image classification neural net. I am using this cost function with a sigmoid output layer and ReLU hidden layers. The neural net has 3 ...
1answer
45 views

### Inefficient Gradient Calculation in Neural Nets

I understood backprop: get gradient wrt a parameter (i.e. the partial derivative) using the chain rule. In the post http://www.offconvex.org/2016/12/20/backprop/ the authors say that the inefficient ...
1answer
375 views

### Backpropagation matrix multiply error Andrew Ng Machine Learning

In the Backpropagation algorithm video(Gradient Computation, week 5), he has taken an example neural network of 4 layers.(Input, 2 Hidden, Output). So, I had made my own example, I have taken <...
2answers
3k views

### Why can RNNs with LSTM units also suffer from “exploding gradients”?

I have a basic knowledge of how RNNs (and, in particular, with LSTMs units) work. I have a pictorial idea of the architecture of an LSTM unit, that is a cell and a few gates, which regulate the flow ...
1answer
202 views

### Inception module backpropagation - how to get the input error?

I'm implementing an Inception module from scratch, specifically this version with dimensionality reduction, and I'm not sure about how to calculate the input error delta when doing the backpropagation ...
1answer
55 views

### NN to fill in blanks for desired output

Assume we have 1000 features that are fed into a NN classifier and that the NN is already trained well. The 1 output neuron has an activation > 0 to indicate the class ...
1answer
811 views

### Should one-hot output encoding be used in backpropagation?

I'm going through the process of writing backpropagation for a neural network. In particular I'm building a MNIST classifier. I'm wondering if it is better to apply the cost function and ...
1answer
614 views

### Changing the weights values in backprop, by how much?

So im reading through this link on Neural Nets and regression and on this page the part about back propagation which reads: To perform backpropagation and make the network learn, you simply compare ...
0answers
127 views

### Minibatch BPTT application in recurrent neural nets

I'm trying to wrap my head around the finer points of backpropagation through time and was hoping to get some clarification on something. For the most part, I understand the general idea of BPTT: we ...
0answers
57 views

### Working out the derivatives in backproagation

So I have no calculus experience what so ever and I've been tasked to build a neural network so finding the derivatives is proving quite problematic with my limited calculus experience. I've got the ...
1answer
572 views

### SGD learning in CNN gets stuck when using a max pooling layer (x-post from DataScience) [closed]

I'm working on a CNN library for a university project and I'm having some trouble implementing the backpropagation through the max pooling layer. Please note that the whole thing was built from ...
0answers
330 views

### Why aren't vanishing gradients for deep networks a problem?

This wikipedia article for Autoencoders states the following [07.30am UTC, 28th November 2017]: An autoencoder is often trained using one of the many variants of backpropagation (such as conjugate ...
1answer
302 views

### Deep NNs, backpropagation and error calculation [duplicate]

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
1answer
441 views

### Deep NNs, backpropagation and error calculation

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
1answer
582 views

### back-propagation derivatives

In the 10th video of week3 of Ng course on Deep Neural Networks in coursera, there is a slide that i attached. Why he used elementwise product (vs normal matrix product) in this slide? Is it only for ...
1answer
27 views

### Back propogation Algorithm dervivation

I am reading the following link : http://blog.manfredas.com/backpropagation-tutorial/ for understanding the derivation of the back propogation algorithm. In the step for computing the partial ...
0answers
148 views

### Is my Neural Network chain rule correct?

Suppose I have the following architecture Where $HA_i$ and $OA_i$ are the activated values of the hidden and output nodes respectively, and $W_i$ are the weights between nodes. I want to find ...
1answer
4k views

### Mean Absolute Error (MAE) derivative

$MAE=|y_{pred} - y_{true}|$ $\dfrac{dMAE}{dy_{pred}} = ?$ I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some ...
1answer
3k views

### Batch normalization: How to update gamma and beta during backpropagation training step?

The backpropagation step of batch normalization computes the derivative of gamma (let's call it dgamma) and the derivative of <...
1answer
468 views

### ANN: Weights grow very large, can I scale them?

I have written an ANN algorithm. And after several iterations my weights grow largely and there's this error which says the value of them is overflowing and therefore the outputs are NaNs. Does it ...
1answer
722 views

### Why zero-centered output affected the backpropagation?

I read the answer in Why are non zero-centered activation functions a problem in backpropagation? but I still can't understand. Assume$$f=\sum w_ix_i+b$$ $$\sigma(x)=\dfrac{1}{1+e^{-x}}$$, and loss ...
1answer
3k views

### Sigmoid activation hurts training a NN on pyTorch

I'm a beginner in the field of Machine Learning and I'm currently trying to get my hands "dirty" for the first time with some code after completing a course in that field. I'm using pyTorch to train ...
0answers
44 views

### What am I missing in designing ANN? [duplicate]

I implemented ANN and my dataset have the first 100 data from class 1, and next 100 from class 2,..., and last 100 from class 10 (so I have 10 binary output units). I do back_propagation on my data to ...
1answer
340 views

### Backpropagation in multi-layer perceptron (MLP) doesn't converge [closed]

My simple fully-connected multilayer perceptron (MLP) that I'm writing for academic purposes is causing to me sleep deprivation. I can't figure out why my MLP learns poorly, even if I try to solve ...
0answers
2k views

### Matrix Backpropagation with Softmax and Cross Entropy

I'm having trouble deriving the matrix form of backpropagation. As an example, let's suppose we have the following network: There are two nodes in the input layer plus a bias node fixed at 1, three ...
0answers
52 views

### Why, empirically, does cross entropy composed with softmax have such a simple derivative? [closed]

Was one function chosen to simplify the derivatives based on the other? There has to be some intelligent design relating to their derivatives, right? The derivative is simply ...
0answers
61 views

### How is a vector differentiation by a matrix defined?

I am a beginner of the machine learning. And at the same time, this is the first time to ask a question in this site, so I might have a problem with this question in terms of understandability. And ...
1answer
467 views

### What is the difference in “weight update process” in gradient descent vs Stochastic gradient descent?

Question In normal GD the weights are updated for every row in the training dataset while in SGD the weights are updated only once for the mini batch based on cummulative dLoss/dw1, dLoss/dw2 . Is my ...
1answer
389 views

### Matrix derivatives from Lecture 4 of CS231n

I've been going through CS231n material. I am confused with this matrix derivation. Here in this image the derivation of df/dx is given. Its from lecture 4 slide 73. I understand this way of ...