Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
0
votes
1answer
62 views

Backpropagation - partial derivatives

I am currently reading the Neural networks and deep learning book by Michael Nielsen. I have a question regarding the backpropagation chapter: Background: He explains the influence of a neuron on ...
4
votes
1answer
1k views

How do Variational Auto Encoders backprop past the sampling step

From my understanding of VAE's, there's a step during training in the middle where, after the encoder produces a mean and standard deviation, random samples are drawn from the given learned ...
0
votes
0answers
28 views

Back-propagation in presence of dropout [duplicate]

Intuitively dropout make sense to me but I don't understand how backpropagation works in presence of dropout. It looks like at each training step we backpropagate gradients to parameters in the ...
0
votes
1answer
530 views

How does backpropagation differ from reverse-mode autodiff

Going through this book, I am familiar with the following: For each training instance the backpropagation algorithm first makes a prediction (forward pass), measures the error, then goes through ...
1
vote
0answers
45 views

Can Regularization by achieved using Relative Sensitivity?

In a Mathematical Model we measure the sensitivity of the output with respect to the parameters and it is desirable that a small change in a parameter doesn't lead to wild fluctuations in the output ...
1
vote
1answer
310 views

How does backpropagation work in the case of reinforcement learning for games?

If we want a neural network to learn how to recognize e.g. digits, the backpropagation procedure is as follows: Let the NN look at an image of a digit, and output its probabilities on the different ...
1
vote
1answer
377 views

how to derive the gradient of batch normalization

I'm trying to figure out the gradient of batch norm wrt x for backprop, but I get stuck in what I will call 'the triangle of (gradient) death'. I present to you the triangle of death (in red), in the ...
4
votes
1answer
680 views

What is the difference among stochastic, batch and mini-batch learning styles?

So far as I know, we have the following scenario: stochastic: The error is calculated for each sample s. So, we can calculate the gradients for s. And we can update the weights of the network ...
1
vote
0answers
798 views

Numerical gradient checking (best practices)

I've implemented a neural network and am using numerical gradient checking to validate the back-propagation algorithm is working correctly. I'm using the standard method to calculate the numerical ...
0
votes
1answer
94 views

Intuitive Explanation for Gradients in ReLU-Activated Networks

When calculating gradients for backpropagation, a surprising result is that the gradient of the weights in a ReLU-activated layer ends up being a diagonal matrix. I am seeking to understand how this ...
29
votes
4answers
16k views

Why is tanh almost always better than sigmoid as an activation function?

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$. The reason he gives is that the outputs using $tanh$ ...
3
votes
1answer
220 views

Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
2
votes
1answer
1k views

Working for Logistic regression partial derivatives

In Andrew Ng's Neural Networks and Deep Learning course on Coursera the logistic regression loss function for a single training example is given as: $$ \mathcal L(a,y) = - \Big(y\log a + (1 - y)\log (...
2
votes
2answers
463 views

Can we use backpropagation to fit other models?

It appears that backpropagation is exclusively used to train neural network models. Why not use it to fit other models. For example - Taylor polynomials: $$ f(x) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)...
12
votes
1answer
1k views

Matrix form of backpropagation with batch normalization

Batch normalization has been credited with substantial performance improvements in deep neural nets. Plenty of material on the internet shows how to implement it on an activation-by-activation basis. ...
0
votes
1answer
1k views

Non zero centered activation functions

I read the following section from cs231n course notes: Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this ...
1
vote
0answers
321 views

finding the loss derivative w.r.t. weights for a convolutional layer

Take the loss function: $$ \mathrm{loss} ~~=~~ \sum_{i=1}^N \left( -z_{}[y] + \log{\left( \sum_{c=1}^{10} \exp(z_{}[c]) \right)} \right)$$ where $z \in \mathbb{R}^{10}$ is the input to the softmax ...
39
votes
1answer
20k views

How is softmax_cross_entropy_with_logits different from softmax_cross_entropy_with_logits_v2?

Specifically, I suppose I wonder about this statement: Future major versions of TensorFlow will allow gradients to flow into the labels input on backprop by default. Which is shown when I ...
9
votes
0answers
3k views

Backpropagation on a convolutional layer

Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually). First let me try ...
0
votes
1answer
84 views

Neural network gives incorrect outputs although its backpropagates correctly

I am trying to program an OCR based on neural networks. It has in sum 3 layers : Input( 32 * 32 ) => Hidden (180) => Output(2) It task is to recognize the numbers 1 and 2. If I let the network train ...
0
votes
1answer
77 views

In Deep Learning, how much does randomness help generalization?

Let's assume at the beginning the NN is initialized with random weights, then the Backpropagation "shapes" the weights but the signal is strongest close to the final layer (where it gets computed) and ...
1
vote
2answers
233 views

Backpropagation gradients don't match approximated gradients

I am in the process of implementing back propagation into my image classification neural net. I am using this cost function with a sigmoid output layer and ReLU hidden layers. The neural net has 3 ...
0
votes
1answer
45 views

Inefficient Gradient Calculation in Neural Nets

I understood backprop: get gradient wrt a parameter (i.e. the partial derivative) using the chain rule. In the post http://www.offconvex.org/2016/12/20/backprop/ the authors say that the inefficient ...
0
votes
1answer
375 views

Backpropagation matrix multiply error Andrew Ng Machine Learning

In the Backpropagation algorithm video(Gradient Computation, week 5), he has taken an example neural network of 4 layers.(Input, 2 Hidden, Output). So, I had made my own example, I have taken <...
12
votes
2answers
3k views

Why can RNNs with LSTM units also suffer from “exploding gradients”?

I have a basic knowledge of how RNNs (and, in particular, with LSTMs units) work. I have a pictorial idea of the architecture of an LSTM unit, that is a cell and a few gates, which regulate the flow ...
1
vote
1answer
202 views

Inception module backpropagation - how to get the input error?

I'm implementing an Inception module from scratch, specifically this version with dimensionality reduction, and I'm not sure about how to calculate the input error delta when doing the backpropagation ...
1
vote
1answer
55 views

NN to fill in blanks for desired output

Assume we have 1000 features that are fed into a NN classifier and that the NN is already trained well. The 1 output neuron has an activation > 0 to indicate the class ...
0
votes
1answer
811 views

Should one-hot output encoding be used in backpropagation?

I'm going through the process of writing backpropagation for a neural network. In particular I'm building a MNIST classifier. I'm wondering if it is better to apply the cost function and ...
0
votes
1answer
614 views

Changing the weights values in backprop, by how much?

So im reading through this link on Neural Nets and regression and on this page the part about back propagation which reads: To perform backpropagation and make the network learn, you simply compare ...
1
vote
0answers
127 views

Minibatch BPTT application in recurrent neural nets

I'm trying to wrap my head around the finer points of backpropagation through time and was hoping to get some clarification on something. For the most part, I understand the general idea of BPTT: we ...
2
votes
0answers
57 views

Working out the derivatives in backproagation

So I have no calculus experience what so ever and I've been tasked to build a neural network so finding the derivatives is proving quite problematic with my limited calculus experience. I've got the ...
1
vote
1answer
572 views

SGD learning in CNN gets stuck when using a max pooling layer (x-post from DataScience) [closed]

I'm working on a CNN library for a university project and I'm having some trouble implementing the backpropagation through the max pooling layer. Please note that the whole thing was built from ...
2
votes
0answers
330 views

Why aren't vanishing gradients for deep networks a problem?

This wikipedia article for Autoencoders states the following [07.30am UTC, 28th November 2017]: An autoencoder is often trained using one of the many variants of backpropagation (such as conjugate ...
2
votes
1answer
302 views

Deep NNs, backpropagation and error calculation [duplicate]

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
4
votes
1answer
441 views

Deep NNs, backpropagation and error calculation

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
0
votes
1answer
582 views

back-propagation derivatives

In the 10th video of week3 of Ng course on Deep Neural Networks in coursera, there is a slide that i attached. Why he used elementwise product (vs normal matrix product) in this slide? Is it only for ...
0
votes
1answer
27 views

Back propogation Algorithm dervivation

I am reading the following link : http://blog.manfredas.com/backpropagation-tutorial/ for understanding the derivation of the back propogation algorithm. In the step for computing the partial ...
1
vote
0answers
148 views

Is my Neural Network chain rule correct?

Suppose I have the following architecture Where $ HA_i$ and $OA_i$ are the activated values of the hidden and output nodes respectively, and $W_i$ are the weights between nodes. I want to find ...
1
vote
1answer
4k views

Mean Absolute Error (MAE) derivative

$MAE=|y_{pred} - y_{true}|$ $\dfrac{dMAE}{dy_{pred}} = ?$ I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some ...
4
votes
1answer
3k views

Batch normalization: How to update gamma and beta during backpropagation training step?

The backpropagation step of batch normalization computes the derivative of gamma (let's call it dgamma) and the derivative of <...
1
vote
1answer
468 views

ANN: Weights grow very large, can I scale them?

I have written an ANN algorithm. And after several iterations my weights grow largely and there's this error which says the value of them is overflowing and therefore the outputs are NaNs. Does it ...
3
votes
1answer
722 views

Why zero-centered output affected the backpropagation?

I read the answer in Why are non zero-centered activation functions a problem in backpropagation? but I still can't understand. Assume$$f=\sum w_ix_i+b$$ $$\sigma(x)=\dfrac{1}{1+e^{-x}}$$, and loss ...
4
votes
1answer
3k views

Sigmoid activation hurts training a NN on pyTorch

I'm a beginner in the field of Machine Learning and I'm currently trying to get my hands "dirty" for the first time with some code after completing a course in that field. I'm using pyTorch to train ...
0
votes
0answers
44 views

What am I missing in designing ANN? [duplicate]

I implemented ANN and my dataset have the first 100 data from class 1, and next 100 from class 2,..., and last 100 from class 10 (so I have 10 binary output units). I do back_propagation on my data to ...
1
vote
1answer
340 views

Backpropagation in multi-layer perceptron (MLP) doesn't converge [closed]

My simple fully-connected multilayer perceptron (MLP) that I'm writing for academic purposes is causing to me sleep deprivation. I can't figure out why my MLP learns poorly, even if I try to solve ...
2
votes
0answers
2k views

Matrix Backpropagation with Softmax and Cross Entropy

I'm having trouble deriving the matrix form of backpropagation. As an example, let's suppose we have the following network: There are two nodes in the input layer plus a bias node fixed at 1, three ...
3
votes
0answers
52 views

Why, empirically, does cross entropy composed with softmax have such a simple derivative? [closed]

Was one function chosen to simplify the derivatives based on the other? There has to be some intelligent design relating to their derivatives, right? The derivative is simply ...
2
votes
0answers
61 views

How is a vector differentiation by a matrix defined?

I am a beginner of the machine learning. And at the same time, this is the first time to ask a question in this site, so I might have a problem with this question in terms of understandability. And ...
1
vote
1answer
467 views

What is the difference in “weight update process” in gradient descent vs Stochastic gradient descent?

Question In normal GD the weights are updated for every row in the training dataset while in SGD the weights are updated only once for the mini batch based on cummulative dLoss/dw1, dLoss/dw2 . Is my ...
0
votes
1answer
389 views

Matrix derivatives from Lecture 4 of CS231n

I've been going through CS231n material. I am confused with this matrix derivation. Here in this image the derivation of df/dx is given. Its from lecture 4 slide 73. I understand this way of ...