# Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

102 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4k views

### Backpropagation on a convolutional layer

Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually). First let me try ...
4k views

### Deriving linear regression gradient with MSE

So I've been tinkering around with the backpropagation algorithm and to try to get a better understanding of how it works and my calculus is quite rusty. I've derived the gradient for linear ...
327 views

### Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:...
574 views

### ReLU derivative - second order effects

I am reading the Deep Learning Book, where there is a section on generalisations of the ReLU (section 6.3.1). It states: The second derivative of the rectifying operation is 0 almost everywhere, ...
136 views

### Backpropagation - computing partial derivative with respect to W

I am following a chapter on backprop derivation from the online book by Michael Nielsen In particular, following equation is derived in Chapter 2: ${∂C\over∂w^{l}_{jk}}=a^{l−1}_{k}δ^{l}_{j}$ Now, I ...
1k views

### Backpropagation proof and usage confusion

I've been taking Andrew Ng's course on Coursera, and although it has been great so far, I loathe his lack of supplementary documents on proofs. Thankfully, there are some great articles found pretty ...
403 views

### Batch-normalization back propagation equation

I have been trying to understand applying the back propagation on batch-normalization. However, the matrix multiplication and equations involved have got me confused and I feel that my understanding ...
122 views

### Dimensions in single layer NN gradient

Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
94 views

### Derive the gradients of a basic neural network

Given a neural network as following \begin{align*} &J = CE(y,\hat{y})=-\sum_i y_i log(\hat{y}_i)\\ &\hat{y} = softmax(z_2)\\ &z_2 = hW_2+b_2\\ &h = sigmoid(z_1)\\ &...
36 views

### How to compare the weight matrices from different gradient descent runs?

I would like to compare the end results of gradient descent for runs which use different initial weights and different orders of training data. I know that index by index in the matrix will make no ...
879 views

### Padding and stride in backpropagation of a conv net

I am trying to implement the back-propagation of a simple convolutional network. Specifically I understand that one of the steps is the convolution of the gradients coming from the next layer, with ...
122 views

### RNNs: backprop loss from just the last time step or every single one?

Consider a simple task that predict the next alphabets based on previous ones using RNNs. That is, during model inference, we would like the model to output y1_hat (...
57 views

### Working out the derivatives in backproagation

So I have no calculus experience what so ever and I've been tasked to build a neural network so finding the derivatives is proving quite problematic with my limited calculus experience. I've got the ...
355 views

### Why aren't vanishing gradients for deep networks a problem?

This wikipedia article for Autoencoders states the following [07.30am UTC, 28th November 2017]: An autoencoder is often trained using one of the many variants of backpropagation (such as conjugate ...
2k views

### Matrix Backpropagation with Softmax and Cross Entropy

I'm having trouble deriving the matrix form of backpropagation. As an example, let's suppose we have the following network: There are two nodes in the input layer plus a bias node fixed at 1, three ...
69 views

### How is a vector differentiation by a matrix defined?

I am a beginner of the machine learning. And at the same time, this is the first time to ask a question in this site, so I might have a problem with this question in terms of understandability. And ...
42 views

### Neural Networks for predicting Energy at particular date

I am trying to predict Solar Energy value at particular date.So,for this I am applying Artificial Neural Networks model.I am having problem in deciding activation function. Since sigmoid function ...
337 views

### When doing backpropagation, why do we multiply with the derivative of the activation function

I was reading about backpropagation, which in my case works like this: I don't understand why when computing the error term, we are multiplying with the derivative of the activation function: Why ...
248 views

### CNN pooling and convolution

In the past, I've worked with the MNIST dataset and I am currently working with Java. I do not use any external libraries like numpy (for python) or something like that. I started from scratch and ...
406 views

### Deriving Gradients for a Vanilla LSTM

I've been banging my head on this for far too long. The following code should be easy to understand; can someone assist me in discovering what I've done wrong? The code passes a numerical gradient ...
89 views

### Implementing backpropagation for dataset preprocessing

I have this algorithm written that can perform the XOR operation. How can i modify it so that it can process the large dataset? ...
535 views

### Scale MNIST-Data to [-0.9, 0.9]

I'm programming a neural network for MNIST-Recognition. My net has a pretty good performance, with accuracy > 98% on test set. But the training is very slow. So I thought it would be faster if I scale ...
800 views

### Fast RTRL(Real Time Recurrent Learning) for RNN

Assume generic RNN has update formula: $\mathbf{h}_{t+1} = f(\mathbf{x_t},\mathbf{h_t},\mathbf{\theta})$  Where $\mathbf{x}$ is input vector, $\mathbf{h}$ is hidden state vector, and $\theta$ is ...
399 views

### Does Adding more neural units reduce the probability of trapping in a local minima?

Consider a multi-layer neural network that learns its weights with backpropagation (and gradient descent). Hence, there is a probability that we trap into a local minimum. Will adding more neural ...
91 views

746 views

### Backpropagation for Bias in Neural Networks

I have a problem in my neural network relating to the bias vector. I'm using this source as a reference. My understanding of calculating the bias is that the partial derivative cost function with ...
94 views

### Under periodic BPTT, is softmax evaluated only at the end of the period?

Suppose I have a continuous sequence $X$ of words and I wish to train a RNN language model. According to , I would split $X$ into subsequences $X^{1..|X|/k_1}$ $k_1$ sized subsequences ($k_1$ is ...
37 views

### Neural Network probabilities converging to biases

I'm creating an Android app which can use a variety of classification formula, and while I have normal Softmax done correctly, I keep having an issue with the Softmax Neural Network. After about 10 ...