Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

102 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
13
votes
0answers
4k views

Backpropagation on a convolutional layer

Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually). First let me try ...
6
votes
0answers
4k views

Deriving linear regression gradient with MSE

So I've been tinkering around with the backpropagation algorithm and to try to get a better understanding of how it works and my calculus is quite rusty. I've derived the gradient for linear ...
5
votes
0answers
327 views

Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:...
4
votes
0answers
574 views

ReLU derivative - second order effects

I am reading the Deep Learning Book, where there is a section on generalisations of the ReLU (section 6.3.1). It states: The second derivative of the rectifying operation is 0 almost everywhere, ...
4
votes
0answers
136 views

Backpropagation - computing partial derivative with respect to W

I am following a chapter on backprop derivation from the online book by Michael Nielsen In particular, following equation is derived in Chapter 2: ${∂C\over∂w^{l}_{jk}}=a^{l−1}_{k}δ^{l}_{j}$ Now, I ...
4
votes
2answers
1k views

Backpropagation proof and usage confusion

I've been taking Andrew Ng's course on Coursera, and although it has been great so far, I loathe his lack of supplementary documents on proofs. Thankfully, there are some great articles found pretty ...
3
votes
0answers
403 views

Batch-normalization back propagation equation

I have been trying to understand applying the back propagation on batch-normalization. However, the matrix multiplication and equations involved have got me confused and I feel that my understanding ...
3
votes
0answers
122 views

Dimensions in single layer NN gradient

Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
3
votes
0answers
94 views

Derive the gradients of a basic neural network

Given a neural network as following \begin{align*} &J = CE(y,\hat{y})=-\sum_i y_i log(\hat{y}_i)\\ &\hat{y} = softmax(z_2)\\ &z_2 = hW_2+b_2\\ &h = sigmoid(z_1)\\ &...
2
votes
0answers
36 views

How to compare the weight matrices from different gradient descent runs?

I would like to compare the end results of gradient descent for runs which use different initial weights and different orders of training data. I know that index by index in the matrix will make no ...
2
votes
1answer
879 views

Padding and stride in backpropagation of a conv net

I am trying to implement the back-propagation of a simple convolutional network. Specifically I understand that one of the steps is the convolution of the gradients coming from the next layer, with ...
2
votes
1answer
122 views

RNNs: backprop loss from just the last time step or every single one?

Consider a simple task that predict the next alphabets based on previous ones using RNNs. That is, during model inference, we would like the model to output y1_hat (...
2
votes
0answers
57 views

Working out the derivatives in backproagation

So I have no calculus experience what so ever and I've been tasked to build a neural network so finding the derivatives is proving quite problematic with my limited calculus experience. I've got the ...
2
votes
0answers
355 views

Why aren't vanishing gradients for deep networks a problem?

This wikipedia article for Autoencoders states the following [07.30am UTC, 28th November 2017]: An autoencoder is often trained using one of the many variants of backpropagation (such as conjugate ...
2
votes
0answers
2k views

Matrix Backpropagation with Softmax and Cross Entropy

I'm having trouble deriving the matrix form of backpropagation. As an example, let's suppose we have the following network: There are two nodes in the input layer plus a bias node fixed at 1, three ...
2
votes
0answers
69 views

How is a vector differentiation by a matrix defined?

I am a beginner of the machine learning. And at the same time, this is the first time to ask a question in this site, so I might have a problem with this question in terms of understandability. And ...
2
votes
0answers
42 views

Neural Networks for predicting Energy at particular date

I am trying to predict Solar Energy value at particular date.So,for this I am applying Artificial Neural Networks model.I am having problem in deciding activation function. Since sigmoid function ...
2
votes
0answers
337 views

When doing backpropagation, why do we multiply with the derivative of the activation function

I was reading about backpropagation, which in my case works like this: I don't understand why when computing the error term, we are multiplying with the derivative of the activation function: Why ...
2
votes
1answer
248 views

CNN pooling and convolution

In the past, I've worked with the MNIST dataset and I am currently working with Java. I do not use any external libraries like numpy (for python) or something like that. I started from scratch and ...
2
votes
0answers
406 views

Deriving Gradients for a Vanilla LSTM

I've been banging my head on this for far too long. The following code should be easy to understand; can someone assist me in discovering what I've done wrong? The code passes a numerical gradient ...
2
votes
0answers
89 views

Implementing backpropagation for dataset preprocessing

I have this algorithm written that can perform the XOR operation. How can i modify it so that it can process the large dataset? ...
2
votes
0answers
535 views

Scale MNIST-Data to [-0.9, 0.9]

I'm programming a neural network for MNIST-Recognition. My net has a pretty good performance, with accuracy > 98% on test set. But the training is very slow. So I thought it would be faster if I scale ...
2
votes
0answers
800 views

Fast RTRL(Real Time Recurrent Learning) for RNN

Assume generic RNN has update formula: $\mathbf{h}_{t+1} = f(\mathbf{x_t},\mathbf{h_t},\mathbf{\theta})$ [1] Where $\mathbf{x}$ is input vector, $\mathbf{h}$ is hidden state vector, and $\theta$ is ...
2
votes
1answer
399 views

Does Adding more neural units reduce the probability of trapping in a local minima?

Consider a multi-layer neural network that learns its weights with backpropagation (and gradient descent). Hence, there is a probability that we trap into a local minimum. Will adding more neural ...
2
votes
0answers
91 views

Regulation term disappears during taking gradient of cost function by using backpropagation

I'm trying to take the derivative of the cost function with respect to parameter $\theta$. The problem is $\frac{dJ(\theta)}{d\theta}$, somehow, is not equal to $\frac{dJ(\theta)}{dz}\cdot \frac{dz}{d\...
2
votes
0answers
108 views

Training a MLP after pretraining RBMs with dropout

Let's say I have a couple of RBMs that I pretrained and that I used dropout. When finetuning, how does having used dropout effect backpropagation? Do I still use dropout while backprogating and change ...
1
vote
0answers
12 views

backpropagation between fully connected layer and convolution layer?

This is a simple example of a network consisting of two convolutional layers and one fully connected layer. ...
1
vote
2answers
241 views

Softmax layer derivative by hand

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=...
1
vote
0answers
21 views

Why is Backpropogation used instead of Rosenblatt's learning Algo or gradient descent to train MLP's?

In roesnblatt's learning algo and gradient descent the output is calculated for each input and based on the error b/w the outputs calculated and desired outputs the weights are updated. Why is ...
1
vote
0answers
101 views

Can you train deep recurrent neural network layer by layer?

Specifically for Gated Recurrent Unit, and say GRU is "layered" via but suppose it's only 2 layers deep for simplicity, and suppose the "total loss" = $L$ = $\sum{l_{t}} = \sum{error(y^{2}_{t})}$ ...
1
vote
0answers
435 views

Need help writing a neural network for a Pokemon battle

I'm trying to write a neural network that's able to select the optimal course of action in a Pokemon battle. In a battle, there are two different types of actions: use one of the four moves known by ...
1
vote
1answer
140 views

For back propagation in neural networks , how do we calculate vector by matrix derivative?

I am following the course deep learning ai by Andrew NG. In course1 week4, 04-06-Forward and Backward Propagation, he calculates backward propagation for layer $l$ in neural networks as follows (a ...
1
vote
1answer
399 views

Over which set of elements should I perform norm clipping of gradients for backpropagation?

I want to normalise the gradients of my multi-layer perceptron in order to avoid the Exploding Gradients Problem, so I thought I would use l2-normalisation but am unsure about how to apply it to the ...
1
vote
0answers
237 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
1
vote
0answers
47 views

Can Regularization by achieved using Relative Sensitivity?

In a Mathematical Model we measure the sensitivity of the output with respect to the parameters and it is desirable that a small change in a parameter doesn't lead to wild fluctuations in the output ...
1
vote
0answers
991 views

Numerical gradient checking (best practices)

I've implemented a neural network and am using numerical gradient checking to validate the back-propagation algorithm is working correctly. I'm using the standard method to calculate the numerical ...
1
vote
0answers
388 views

finding the loss derivative w.r.t. weights for a convolutional layer

Take the loss function: $$ \mathrm{loss} ~~=~~ \sum_{i=1}^N \left( -z_{}[y] + \log{\left( \sum_{c=1}^{10} \exp(z_{}[c]) \right)} \right)$$ where $z \in \mathbb{R}^{10}$ is the input to the softmax ...
1
vote
0answers
140 views

Minibatch BPTT application in recurrent neural nets

I'm trying to wrap my head around the finer points of backpropagation through time and was hoping to get some clarification on something. For the most part, I understand the general idea of BPTT: we ...
1
vote
0answers
168 views

Is my Neural Network chain rule correct?

Suppose I have the following architecture Where $ HA_i$ and $OA_i$ are the activated values of the hidden and output nodes respectively, and $W_i$ are the weights between nodes. I want to find ...
1
vote
0answers
87 views

Interpretations of chain rule for backprop

In Goodfellow et al.'s Deep Learning, the authors write on page 203: Let $w \in \mathbb{R}$ be the input to the graph. We use the same function $f: \mathbb{R} \rightarrow \mathbb{R}$ as the ...
1
vote
1answer
267 views

Are momentum and (mini-) batch training compatible?

I had a backpropagation model which worked perfectly fine, however I wanted to implemented batch training. Code before batch training (in backpropagation function), pseudocode: ...
1
vote
1answer
50 views

Replicator Neuron Network back propagation problem

Recently I started reading this paper: Link, and found it quite interesting, so I start to write the code Python. The code run with no error, however, the result is not like my expectation. After a ...
1
vote
0answers
299 views

Recursive backpropagation vs backpropagation

I recently read a paper that tried to find out what happened in a RNN by linearization around slow and fixed points. I can't figure out why we have to use linearization around these points. After I ...
1
vote
0answers
33 views

does bp need to be simultanious? - XOR learning problem

I have a problem with XOR learning. I have 2(inputs),2,1(output) neuron neural network with sigmoidal function and normal sets ...
1
vote
0answers
70 views

Elaboration on weight change for output layer in neural network

I have trouble understanding the section regarding backpropagation in Murphy's Machine Learning book. He derives the weight change for the output layer (16.68) as follows: $\nabla_{\mathbf{w}_k} J_n ...
1
vote
0answers
746 views

Backpropagation for Bias in Neural Networks

I have a problem in my neural network relating to the bias vector. I'm using this source as a reference. My understanding of calculating the bias is that the partial derivative cost function with ...
1
vote
1answer
94 views

Under periodic BPTT, is softmax evaluated only at the end of the period?

Suppose I have a continuous sequence $X$ of words and I wish to train a RNN language model. According to [1], I would split $X$ into subsequences $X^{1..|X|/k_1}$ $k_1$ sized subsequences ($k_1$ is ...
1
vote
0answers
37 views

Neural Network probabilities converging to biases

I'm creating an Android app which can use a variety of classification formula, and while I have normal Softmax done correctly, I keep having an issue with the Softmax Neural Network. After about 10 ...
1
vote
0answers
83 views

how does a neural network with stochastic backpropagation make sure it doesn't “undo” previous learning?

Assume we have a neural network with stochastic gradient descent used for backpropagation, and therefore each element in the training set is used once to calculate the error, and then to adjust the ...
1
vote
1answer
205 views

Backpropagation Through Time Error Computation

I'm attempting to work through the backpropagation through time terms using this source: http://www.deeplearningbook.org/contents/rnn.html The final formulas are given on pages 385 and 386, but I ...