Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
2
votes
0answers
28 views

Softmax in last layer - error rises but when using sigmoid error decreases

I wrote a neural network from scratch in Python. It has 1 hidden layer which uses tanh activation function. I train it on Iris and MNIST datasets. When I use Sigmoid in the last layer results are very ...
0
votes
1answer
522 views

Calculating Softmax derivative independent of cost function

Note: there is a nearly identical question on Stack Overflow. However, I seem to be missing something... or maybe it's just that Python isn't my first language ;) For a neural network library, I've ...
0
votes
0answers
16 views

pooling variable depth samplewise / are there any importantce to take into account?

My purpose is to apply pooling strategy (max/average) with a variable depth length of the samples in a batch. My question is whether is there something what I should take into account. Does it ...
0
votes
1answer
602 views

CNN convolutional layer backpropagation formulas

I tried to implement a CNN in Java but I am stuck at updating the weights in my convolution layer. I tried to create the following image that shows how I calculate each weight delta and error signals:...
0
votes
0answers
8 views

Backpropagation through an average gate

I'm currently going through the CS231n: CNN's for Computer Vision course offered for free by Stanford University and had a question regarding one part of the assignment. I'm currently trying to ...
28
votes
7answers
24k views

Danger of setting all initial weights to zero in Backpropagation

Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?
0
votes
1answer
337 views

Recurrent neural networks - why is the vanishing gradient a problem?

I understand why the gradient tends to become small in the early layers of a deep network. However I am trying to clarify why this is a problem in the case of RNNs. This is what I understand: the ...
1
vote
1answer
310 views

Over which set of elements should I perform norm clipping of gradients for backpropagation?

I want to normalise the gradients of my multi-layer perceptron in order to avoid the Exploding Gradients Problem, so I thought I would use l2-normalisation but am unsure about how to apply it to the ...
25
votes
1answer
10k views

Why are non zero-centered activation functions a problem in backpropagation?

I read here the following: Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this soon) would be receiving ...
0
votes
0answers
11 views

Backpropagation through 2D transposed convolution layer

I’m looking for an explanation for the backwards pass in a conv2d transpose layer. My main problem is that the deltas from the next layer are larger than the input of the previous layer. Hence, I can’...
3
votes
1answer
139 views

Backpropagation in capsule networks

Trying to create a capsule network implementation, I've browsed through several tutorials and code sources, but was unable to find how back-propagation for capsule networks is implemented. It is not ...
0
votes
0answers
15 views

Computing derivatives for backpropagation across a convolution step

This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and ...
29
votes
4answers
16k views

Why is tanh almost always better than sigmoid as an activation function?

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$. The reason he gives is that the outputs using $tanh$ ...
3
votes
1answer
959 views

Backpropagation proof and usage confusion

I've been taking Andrew Ng's course on Coursera, and although it has been great so far, I loathe his lack of supplementary documents on proofs. Thankfully, there are some great articles found pretty ...
0
votes
1answer
98 views

Multiple filters during backpropagation in convolutional neural network

Let's say we have an Input 10x10x3 (WxHxD) and 5 filters 3x3x3. Convolution between Input and filters will be 8x8x5. During backpropagation we will get error with the same size 8x8x5. While ...
1
vote
1answer
216 views

Vanishing gradient in basic 3-layer neural networks?

A 3-layer network has two layers of connections (between input and hidden layers and between hidden and output layer). Doesn't this mean that the gradient "vanishes", at least slighty, when training ...
0
votes
0answers
20 views

Correct gradient with custom weight update

I have a layer $f_{(a,b)}$, where $(a,b)$ are some parameters. During training, $(a,b)$ get updated using a custom update-scheme $g$. The thing is that $(a,b)$ don't get updated during the forward-...
0
votes
1answer
249 views

Are momentum and (mini-) batch training compatible?

I had a backpropagation model which worked perfectly fine, however I wanted to implemented batch training. Code before batch training (in backpropagation function), pseudocode: ...
0
votes
0answers
23 views

Faulty backpropagation of hidden layers? (C#)

I have been getting into machine learning and I decided to create my own NN Backpropagation program without libraries. I was making progress quite nicely for a while, but I got stuck and I can't seem ...
2
votes
1answer
238 views

CNN pooling and convolution

In the past, I've worked with the MNIST dataset and I am currently working with Java. I do not use any external libraries like numpy (for python) or something like that. I started from scratch and ...
3
votes
1answer
103 views

How exactly is the error backpropagated in backpropagation?

I am reading a book on neural networks, and am now doing a chapter on backpropagation. (See chapter here). In this chapter, the writer is presenting four equations, that together form the backbone of ...
0
votes
0answers
9 views

Why cache gradients for params between training examples?

I was going through Karpathy's guide here where he defines an simple multiplication gate's forward and backward passes like so ...
0
votes
0answers
14 views

Difference between the convolution and correlation backpropagation

In the article about the convolution backpropagation, the computation of gradients to the input needs to rotate the weight and the computation of gradients to the weight also needs to rotate the input....
0
votes
1answer
26 views

2D max pool gradient propagation

I am trying to understand gradient propagation for a 2D max pooling operation when there is multiple filters for each position in the 2D grid (i.e. size = $b\times2\times2\times d$, where $b$ is ...
0
votes
0answers
32 views

Why doesn't my Feed-forward NN work when I try to train it with multiple inputs? [duplicate]

Basically I am trying to create a Neural Network in c# from scratch, without using any libraries. The issue that I am facing with right now is whenever I try to train my network with different inputs ...
0
votes
1answer
374 views

Backpropagation matrix multiply error Andrew Ng Machine Learning

In the Backpropagation algorithm video(Gradient Computation, week 5), he has taken an example neural network of 4 layers.(Input, 2 Hidden, Output). So, I had made my own example, I have taken <...
2
votes
1answer
376 views

Does Adding more neural units reduce the probability of trapping in a local minima?

Consider a multi-layer neural network that learns its weights with backpropagation (and gradient descent). Hence, there is a probability that we trap into a local minimum. Will adding more neural ...
0
votes
1answer
23 views

Binarizing Data in a Network using the sign function

I often see the use of the sign function in machine learning models as a way to binarize data (see eqn 1 here for an example). But the derivative of the sign function is the dirac delta function, so ...
2
votes
2answers
269 views

Backpropagating a Dueling Architecture Network: Gradient Calculation

I started coding a Dueling Network Architectures for Deep Reinforcement Learning. I devided my network into two streams, arriving at a V(s) value and A(s,a) values. I arrived at the Q(s,a) output ...
0
votes
0answers
16 views

How to compute weight change for hidden layers with cross-entropy loss? [duplicate]

I'm trying to train a neural net with 1 hidden layer (RELU) softmax output layer cross-entropy loss stochastic gradient descent My implementation seems to work fine when I don't use any hidden ...
0
votes
0answers
24 views

What's the bug in my implementation/understanding of backpropagation?

For learning purposes, I'm trying to implement a simple neural network with only linear layers followed by logistic activation. As far as I understand, the backpropagation algorithm exploits the ...
0
votes
2answers
184 views

Softmax layer derivative by hand

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=...
2
votes
1answer
654 views

Padding and stride in backpropagation of a conv net

I am trying to implement the back-propagation of a simple convolutional network. Specifically I understand that one of the steps is the convolution of the gradients coming from the next layer, with ...
0
votes
1answer
31 views

Backpropagation - Assumption or Error in Bishop's “PRaML”

I am currently reading Bishop - "Pattern Recognition and Machine Learning" (2006) and I could not figure out if I missed an assumption he made or if it is just wrong. In the chapter about ...
0
votes
1answer
300 views

How well should backpropagation agree with finite difference methods when calculating derivatives of the error function?

I have attempted to write a Neural Network code, and it was suggested in my textbook (Bishop - Pattern Recognition & Machine Learning) that a very useful debugging technique is to check your $\...
1
vote
1answer
184 views

Backpropagation Through Time Error Computation

I'm attempting to work through the backpropagation through time terms using this source: http://www.deeplearningbook.org/contents/rnn.html The final formulas are given on pages 385 and 386, but I ...
0
votes
1answer
29 views

Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point: To calculate the gradients, you need to run the .backward method on a Variable, ...
0
votes
1answer
35 views

Softmax with Cross Entropy optimization vs Backpropagation

I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example). The code from the tutorial states "First we need to define the ...
2
votes
1answer
78 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
4
votes
1answer
637 views

What is the most efficient method to handle long time sequences (LSTM)?

I am using LSTM and I have several long time sequences of varying length. Most of them are about 6,000-7,000 timesteps on average, but several are around 40,000 long. I am not sure which of this would ...
1
vote
1answer
19 views

Training Perceptrons with Backprop

Is it possible to train a simple perceptron with a threshold activation function such as this one: https://en.wikipedia.org/wiki/Perceptron with Backpropagation instead of the perceptron rule? is it ...
0
votes
1answer
22 views

Back propagation is done with each batch in a convolutional net, but is it also done with the validation set?

It's my understanding that the weights are updated in a convolutional neural network with each evaluation of a batch. But when the training data has been processed and it comes to predicting ...
3
votes
1answer
358 views

Backprop: Calc gradients for embed layers and how to do gradient check

First I want to check is I understand correctly how backprop works for NN with embed layers. Lets create simplest NN with embed layer Input Layer [i1; i2; i3; i4] - input nodes Embed layer [h1] [h2] ...
1
vote
1answer
467 views

What is the difference in “weight update process” in gradient descent vs Stochastic gradient descent?

Question In normal GD the weights are updated for every row in the training dataset while in SGD the weights are updated only once for the mini batch based on cummulative dLoss/dw1, dLoss/dw2 . Is my ...
0
votes
3answers
46 views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
0
votes
1answer
34 views

Confusion when Learning Parameters in BAYESIAN MODELS

I'm learning Bayesian Models but i still have some issues with the training of the parameters. These are my two questions : 1) Recall the Bayesian formula : $$p(\theta|X) = \frac{ p(X|\theta) \; p(\...
9
votes
1answer
3k views

Gradients for skipgram word2vec

I am going through the problems in the Stanford NLP deep learning class's written assignment problems http://cs224d.stanford.edu/assignment1/assignment1_soln I am trying to understand the answer for ...
0
votes
1answer
20 views

Is backpropagation is used in validation data set? [closed]

Hello guys I am very confused as I am building a deep learning image classifier from raw python code ,so my question is that:-is backpropagation used in validation set to get the model more accuracy ...
-1
votes
3answers
513 views

Difference between cross-validation and back propagation

sorry if this is a dumb question. I want to know what the difference is? Mainly what how they're objectives differ from each other? I know that in cross validation you divide the training set into ...
1
vote
1answer
60 views

For back propagation in neural networks , how do we calculate vector by matrix derivative?

I am following the course deep learning ai by Andrew NG. In course1 week4, 04-06-Forward and Backward Propagation, he calculates backward propagation for layer $l$ in neural networks as follows (a ...