Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
0
votes
1answer
29 views

How does Cross-Entropy (log loss) work with backpropagation?

I am having some trouble understanding how Cross Entropy would work with backpropagation. For backpropagation we exploit the chain rule to find the partial derivative of the Error function in terms of ...
0
votes
0answers
30 views

Why backpropagation if loss function is not convex in nature?

Backpropagation contains the method of gradient decent, which works well for convex loss functions with a global minima. But, for training, in most of the neural network tasks, backpropagation is ...
0
votes
1answer
26 views

Why Backpropagation is in negative

New to Backpropagation. In this example here and in all other example we calculate new weights by finding the derivative of the sigmoid https://repl.it/@vzhou842/An-Introduction-to-Neural-Networks ...
0
votes
0answers
37 views

Backprop through un-differentiable function f(x)?

Let's say I have a vector of gradients (ie. dL / dy). These gradients are the result of taking the derivative of the Loss function with respect to the output of a ...
0
votes
0answers
14 views

CNN Backpropagation Clarrification

Hi I am just trying to make sure my understanding of backpropagation with CNNs is correct, specifically CNNs that have multiple filters in each layer. This is how I have implemented backpropagation ...
0
votes
1answer
18 views

problem with vanishing/exploding gradient problems solution

I have few doubts around vanishing/exploding gradients. The problem with vanishing gradient is, When the weights are randomly initialized in a deep network, During back propagation initial layers ...
1
vote
1answer
49 views

Softmax in last layer - error rises but when using sigmoid error decreases [closed]

I wrote a neural network from scratch in Python. It has 1 hidden layer which uses tanh activation function. I train it on Iris and MNIST datasets. When I use Sigmoid in the last layer results are very ...
0
votes
0answers
18 views

pooling variable depth samplewise / are there any importantce to take into account?

My purpose is to apply pooling strategy (max/average) with a variable depth length of the samples in a batch. My question is whether is there something what I should take into account. Does it ...
0
votes
0answers
8 views

Backpropagation through an average gate

I'm currently going through the CS231n: CNN's for Computer Vision course offered for free by Stanford University and had a question regarding one part of the assignment. I'm currently trying to ...
0
votes
0answers
13 views

Backpropagation through 2D transposed convolution layer

I’m looking for an explanation for the backwards pass in a conv2d transpose layer. My main problem is that the deltas from the next layer are larger than the input of the previous layer. Hence, I can’...
0
votes
0answers
15 views

Computing derivatives for backpropagation across a convolution step

This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and ...
0
votes
0answers
20 views

Correct gradient with custom weight update

I have a layer $f_{(a,b)}$, where $(a,b)$ are some parameters. During training, $(a,b)$ get updated using a custom update-scheme $g$. The thing is that $(a,b)$ don't get updated during the forward-...
0
votes
0answers
24 views

Faulty backpropagation of hidden layers? (C#)

I have been getting into machine learning and I decided to create my own NN Backpropagation program without libraries. I was making progress quite nicely for a while, but I got stuck and I can't seem ...
0
votes
0answers
10 views

Why cache gradients for params between training examples?

I was going through Karpathy's guide here where he defines an simple multiplication gate's forward and backward passes like so ...
0
votes
0answers
18 views

Difference between the convolution and correlation backpropagation

In the article about the convolution backpropagation, the computation of gradients to the input needs to rotate the weight and the computation of gradients to the weight also needs to rotate the input....
3
votes
1answer
110 views

How exactly is the error backpropagated in backpropagation?

I am reading a book on neural networks, and am now doing a chapter on backpropagation. (See chapter here). In this chapter, the writer is presenting four equations, that together form the backbone of ...
0
votes
1answer
27 views

2D max pool gradient propagation

I am trying to understand gradient propagation for a 2D max pooling operation when there is multiple filters for each position in the 2D grid (i.e. size = $b\times2\times2\times d$, where $b$ is ...
0
votes
0answers
32 views

Why doesn't my Feed-forward NN work when I try to train it with multiple inputs? [duplicate]

Basically I am trying to create a Neural Network in c# from scratch, without using any libraries. The issue that I am facing with right now is whenever I try to train my network with different inputs ...
0
votes
1answer
23 views

Binarizing Data in a Network using the sign function

I often see the use of the sign function in machine learning models as a way to binarize data (see eqn 1 here for an example). But the derivative of the sign function is the dirac delta function, so ...
0
votes
0answers
16 views

How to compute weight change for hidden layers with cross-entropy loss? [duplicate]

I'm trying to train a neural net with 1 hidden layer (RELU) softmax output layer cross-entropy loss stochastic gradient descent My implementation seems to work fine when I don't use any hidden ...
0
votes
0answers
26 views

What's the bug in my implementation/understanding of backpropagation?

For learning purposes, I'm trying to implement a simple neural network with only linear layers followed by logistic activation. As far as I understand, the backpropagation algorithm exploits the ...
0
votes
1answer
31 views

Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point: To calculate the gradients, you need to run the .backward method on a Variable, ...
0
votes
1answer
39 views

Softmax with Cross Entropy optimization vs Backpropagation

I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example). The code from the tutorial states "First we need to define the ...
2
votes
1answer
101 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
1
vote
1answer
19 views

Training Perceptrons with Backprop

Is it possible to train a simple perceptron with a threshold activation function such as this one: https://en.wikipedia.org/wiki/Perceptron with Backpropagation instead of the perceptron rule? is it ...
0
votes
1answer
22 views

Back propagation is done with each batch in a convolutional net, but is it also done with the validation set?

It's my understanding that the weights are updated in a convolutional neural network with each evaluation of a batch. But when the training data has been processed and it comes to predicting ...
0
votes
1answer
35 views

Confusion when Learning Parameters in BAYESIAN MODELS

I'm learning Bayesian Models but i still have some issues with the training of the parameters. These are my two questions : 1) Recall the Bayesian formula : $$p(\theta|X) = \frac{ p(X|\theta) \; p(\...
0
votes
3answers
49 views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
0
votes
1answer
22 views

Is backpropagation is used in validation data set? [closed]

Hello guys I am very confused as I am building a deep learning image classifier from raw python code ,so my question is that:-is backpropagation used in validation set to get the model more accuracy ...
0
votes
0answers
16 views

XOR Neural Network, Problem finding shapes of delta for backpropagation algorithm

I am taking the Machine Learning course by Andrew Ng on coursera. I am trying to make a neural network learn to do XOR, but I am facing a problem regarding the shapes of the $\delta$ vectors, and $\...
2
votes
1answer
30 views

Calculating the Policy Gradient for a Monte Carlo REINFORCE Algorithm

I am currently trying to implement the Monte Carlo REINFORCE algorithm, as described in Sutton and Barto's book Reinforcement Learning (p. 328, Second Edition). If $\theta$ denotes the parameter for ...
5
votes
2answers
199 views

Can $\sin(x)$ be used as activation in deep learning?

$\sin(x)$ seems to zero centered which is a desirable property for activation functions. Even the gradient won't vanish at any point. I am not sure if the oscillating nature of the function or its ...
0
votes
0answers
31 views

How to handle maxpool layer backpropagation with recurring max values in same position

Say I have a layer a: 3 4 2 1 5 0 8 6 4 The maxpool using 2x2 filter is: <...
0
votes
1answer
41 views

Meaning of “backpropagate through Gaussian distributions”?

I just started reading about GAN theory properly for the first time and I have a question about a comment in the original GAN paper. On page two there's a paragraph that states the following: ... ...
1
vote
1answer
13 views

How does backpropagation work with mixed architectures?

This question raised to me since I'm unsucessfully training a CNN-LSTM network atm. If for instance, LSTM requires a different type of BP algorithm (TBPTT), how do softwares deal with it? What is the ...
0
votes
0answers
14 views

Clarification on notation used to present back propagation algorithm in 'The Deep Learning Book'

In the deep learning book (free version is available online) the backpropation algorithm is explained in section 6.5. I have a question on equation (6.53): $$\frac{\partial u^{(n)}}{\partial u^{(j)}}...
0
votes
1answer
73 views

How is the loss(Backpropagation) for simple RNN calculated when dealing with batch?

I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself. However, it seems I have not ...
0
votes
1answer
31 views

Backpropagation - Assumption or Error in Bishop's “PRaML”

I am currently reading Bishop - "Pattern Recognition and Machine Learning" (2006) and I could not figure out if I missed an assumption he made or if it is just wrong. In the chapter about ...
0
votes
0answers
261 views

backpropagation for multiple hidden layers

I'm currently implementing a small neural network library to learn the concepts coming with them better. As probably most people do, I'm struggling a bit with the backpropagation algorithm, especially ...
0
votes
0answers
31 views

Training convolutional neural network

I implemented Convolutional Neural Network from scratch for image recognition for 5 classes. When I train it using only one image from one class it seems to be working, because accuracy for this class ...
0
votes
0answers
52 views

Calculation of backpropagation of a NN with skip nodes

Assume this neural net. In our course we had to provide the forward and backward pass for this net. This is the solution we received: Even with the solutions provided I still have problems ...
0
votes
0answers
31 views

What are the advantage of using Sigmoid and Softmax and disadvantageous of both? [duplicate]

I am trying to understand the architecture of the neural network. I am supposed to make decision of how many hidden layers and what activation functions to use in the hidden layer and which activation ...
0
votes
0answers
17 views

ANN for Boundary Value Problem

I have a question regarding solving Boundary Value Problems (BVP) using ANNs. My understanding is that this is currently a challenging task. Most scientific literature on the subject is interested in ...
0
votes
2answers
186 views

Softmax layer derivative by hand

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=...
0
votes
1answer
102 views

Multiple filters during backpropagation in convolutional neural network

Let's say we have an Input 10x10x3 (WxHxD) and 5 filters 3x3x3. Convolution between Input and filters will be 8x8x5. During backpropagation we will get error with the same size 8x8x5. While ...
2
votes
2answers
142 views

How does training affect the norm of weight matrices?

I have a neural network $F(W,x): \mathbb{R}^d \rightarrow \mathbb{R}^k$ with $L$ layers, $m$ neurons per layer, ReLu activation, softmax on the last layer and $n$ datapoint. My loss function is the ...
1
vote
1answer
113 views

Derivative of the loss function w.r.t to X for the backpropagation

I would like to ask you why do we need to calculate a derivative of the loss function w.r.t X? It seems like, that for the backpropagation we need to calculate only a derivative w.r.t W. Can you ...
0
votes
0answers
37 views

Backpropagation through LSTM and MLP

For didactic reason, I am currently implementing in numpy an LSTM network for classifications. I need to add on top of the LSTM another fully connected layer, because I don't want the output to have ...
0
votes
1answer
31 views

Neural Networks - Back Propagation and Perceptrons

While studying about neural networks (still on basics - not Deep Learning etc.,) two questions came on my mind. What is the reason for replacing the hard limiter function in the nodes of the ...
0
votes
0answers
36 views

Mini Batch Gradient Descent Backpropapagation

I am a beginner to machine learning. I have derived the equations for backpropagation, and for the weight update for hidden layers, the update rule uses the output vector of the layer to multiply with ...