# Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

347 questions
Filter by
Sorted by
Tagged with
29 views

### How does Cross-Entropy (log loss) work with backpropagation?

I am having some trouble understanding how Cross Entropy would work with backpropagation. For backpropagation we exploit the chain rule to find the partial derivative of the Error function in terms of ...
30 views

### Why backpropagation if loss function is not convex in nature?

Backpropagation contains the method of gradient decent, which works well for convex loss functions with a global minima. But, for training, in most of the neural network tasks, backpropagation is ...
26 views

### Why Backpropagation is in negative

New to Backpropagation. In this example here and in all other example we calculate new weights by finding the derivative of the sigmoid https://repl.it/@vzhou842/An-Introduction-to-Neural-Networks ...
37 views

### Backprop through un-differentiable function f(x)?

Let's say I have a vector of gradients (ie. dL / dy). These gradients are the result of taking the derivative of the Loss function with respect to the output of a ...
14 views

### CNN Backpropagation Clarrification

Hi I am just trying to make sure my understanding of backpropagation with CNNs is correct, specifically CNNs that have multiple filters in each layer. This is how I have implemented backpropagation ...
18 views

### problem with vanishing/exploding gradient problems solution

I have few doubts around vanishing/exploding gradients. The problem with vanishing gradient is, When the weights are randomly initialized in a deep network, During back propagation initial layers ...
49 views

### Softmax in last layer - error rises but when using sigmoid error decreases [closed]

I wrote a neural network from scratch in Python. It has 1 hidden layer which uses tanh activation function. I train it on Iris and MNIST datasets. When I use Sigmoid in the last layer results are very ...
18 views

### pooling variable depth samplewise / are there any importantce to take into account?

My purpose is to apply pooling strategy (max/average) with a variable depth length of the samples in a batch. My question is whether is there something what I should take into account. Does it ...
8 views

### Backpropagation through an average gate

I'm currently going through the CS231n: CNN's for Computer Vision course offered for free by Stanford University and had a question regarding one part of the assignment. I'm currently trying to ...
13 views

### Backpropagation through 2D transposed convolution layer

I’m looking for an explanation for the backwards pass in a conv2d transpose layer. My main problem is that the deltas from the next layer are larger than the input of the previous layer. Hence, I can’...
15 views

### Computing derivatives for backpropagation across a convolution step

This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and ...
20 views

### Correct gradient with custom weight update

I have a layer $f_{(a,b)}$, where $(a,b)$ are some parameters. During training, $(a,b)$ get updated using a custom update-scheme $g$. The thing is that $(a,b)$ don't get updated during the forward-...
24 views

### Faulty backpropagation of hidden layers? (C#)

I have been getting into machine learning and I decided to create my own NN Backpropagation program without libraries. I was making progress quite nicely for a while, but I got stuck and I can't seem ...
10 views

### Why cache gradients for params between training examples?

I was going through Karpathy's guide here where he defines an simple multiplication gate's forward and backward passes like so ...
18 views

### Difference between the convolution and correlation backpropagation

In the article about the convolution backpropagation, the computation of gradients to the input needs to rotate the weight and the computation of gradients to the weight also needs to rotate the input....
110 views

### How exactly is the error backpropagated in backpropagation?

I am reading a book on neural networks, and am now doing a chapter on backpropagation. (See chapter here). In this chapter, the writer is presenting four equations, that together form the backbone of ...
27 views

### 2D max pool gradient propagation

I am trying to understand gradient propagation for a 2D max pooling operation when there is multiple filters for each position in the 2D grid (i.e. size = $b\times2\times2\times d$, where $b$ is ...
32 views

### Why doesn't my Feed-forward NN work when I try to train it with multiple inputs? [duplicate]

Basically I am trying to create a Neural Network in c# from scratch, without using any libraries. The issue that I am facing with right now is whenever I try to train my network with different inputs ...
23 views

### Binarizing Data in a Network using the sign function

I often see the use of the sign function in machine learning models as a way to binarize data (see eqn 1 here for an example). But the derivative of the sign function is the dirac delta function, so ...
16 views

### How to compute weight change for hidden layers with cross-entropy loss? [duplicate]

I'm trying to train a neural net with 1 hidden layer (RELU) softmax output layer cross-entropy loss stochastic gradient descent My implementation seems to work fine when I don't use any hidden ...
26 views

### What's the bug in my implementation/understanding of backpropagation?

For learning purposes, I'm trying to implement a simple neural network with only linear layers followed by logistic activation. As far as I understand, the backpropagation algorithm exploits the ...
31 views

### Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point: To calculate the gradients, you need to run the .backward method on a Variable, ...
39 views

### Softmax with Cross Entropy optimization vs Backpropagation

I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example). The code from the tutorial states "First we need to define the ...
101 views

### Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
19 views

### Training Perceptrons with Backprop

Is it possible to train a simple perceptron with a threshold activation function such as this one: https://en.wikipedia.org/wiki/Perceptron with Backpropagation instead of the perceptron rule? is it ...
22 views

### Back propagation is done with each batch in a convolutional net, but is it also done with the validation set?

It's my understanding that the weights are updated in a convolutional neural network with each evaluation of a batch. But when the training data has been processed and it comes to predicting ...
35 views

73 views

### How is the loss(Backpropagation) for simple RNN calculated when dealing with batch?

I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself. However, it seems I have not ...
31 views

### Backpropagation - Assumption or Error in Bishop's “PRaML”

I am currently reading Bishop - "Pattern Recognition and Machine Learning" (2006) and I could not figure out if I missed an assumption he made or if it is just wrong. In the chapter about ...
261 views

### backpropagation for multiple hidden layers

I'm currently implementing a small neural network library to learn the concepts coming with them better. As probably most people do, I'm struggling a bit with the backpropagation algorithm, especially ...
31 views

### Training convolutional neural network

I implemented Convolutional Neural Network from scratch for image recognition for 5 classes. When I train it using only one image from one class it seems to be working, because accuracy for this class ...
52 views

### Calculation of backpropagation of a NN with skip nodes

Assume this neural net. In our course we had to provide the forward and backward pass for this net. This is the solution we received: Even with the solutions provided I still have problems ...
31 views

### What are the advantage of using Sigmoid and Softmax and disadvantageous of both? [duplicate]

I am trying to understand the architecture of the neural network. I am supposed to make decision of how many hidden layers and what activation functions to use in the hidden layer and which activation ...
17 views

### ANN for Boundary Value Problem

I have a question regarding solving Boundary Value Problems (BVP) using ANNs. My understanding is that this is currently a challenging task. Most scientific literature on the subject is interested in ...
186 views

### Softmax layer derivative by hand

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=...
102 views

### Multiple filters during backpropagation in convolutional neural network

Let's say we have an Input 10x10x3 (WxHxD) and 5 filters 3x3x3. Convolution between Input and filters will be 8x8x5. During backpropagation we will get error with the same size 8x8x5. While ...
142 views

### How does training affect the norm of weight matrices?

I have a neural network $F(W,x): \mathbb{R}^d \rightarrow \mathbb{R}^k$ with $L$ layers, $m$ neurons per layer, ReLu activation, softmax on the last layer and $n$ datapoint. My loss function is the ...
113 views

### Derivative of the loss function w.r.t to X for the backpropagation

I would like to ask you why do we need to calculate a derivative of the loss function w.r.t X? It seems like, that for the backpropagation we need to calculate only a derivative w.r.t W. Can you ...
37 views

### Backpropagation through LSTM and MLP

For didactic reason, I am currently implementing in numpy an LSTM network for classifications. I need to add on top of the LSTM another fully connected layer, because I don't want the output to have ...