Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
0
votes
1answer
19 views

Interpreting cost change plot in a neural network for learning XOR

I tried to build a neural net for learning XOR. The design is as follows: 1st layer: compute linear function of input 4:2 with 2:2 weights and adding 1:2 bias. 2nd layer: apply sigmoid to all ...
0
votes
2answers
25 views

Neural network training: skipping layers when performing weight updates

Suppose you had many models that are trained to classify a sequence of logical characters into 'satisfiable' or 'unsatisfiable' (e.g., the sequence "A and notA" is unsatisfiable). Suppose all of these ...
0
votes
0answers
18 views

Backpropagation for Linear Softmax classifier

I'm currently implementing a Linear Softmax classifier from scratch where $\mathbf{\hat y} =\mathbf{x^TW}$. I'm not sure about the backpropagation step, however. $L$ denotes the Cross Entropy loss ...
0
votes
0answers
26 views

Some questions about CNN's and the calculation of the gradients

I have succesfully created my own MLP-Library and now I wanted to move on to CNN's. I have already read alot of websites but there are still some things I don't understand (please excuse my English as ...
1
vote
1answer
61 views

How is the learning process for a NN implemented?

Thanks to BP and some scoring/loss functions, a NN can train to do the job as demanded. Or at least try to do so. I think I've understood BP but still I wonder about the following: Let's say there is ...
0
votes
1answer
30 views

Taking derivative for RNN back propogation

I am trying to understand the derivation of backpropagation for recurrent neural networks (RNNs) from this source: https://github.com/go2carter/nn-learn/blob/master/grad-deriv-tex/rnn-grad-deriv.pdf ...
0
votes
0answers
28 views

Backpropagation of hinge loss with L2 regularization

I am trying to figure out the derivatives of multi-class linear model with hinge loss and L2 regularization: $L(x,y) = max(0, 1 - (\sum_{i}{x_{i}} W_{i, t} + b_{t}) + (\sum_{i}{x_{i}} W_{i, k} + b {k}...
0
votes
0answers
10 views

Batched version of gradient calculation

I would like to implement following equation, which sums up the gradients of the output of sample $x_{i}$ wrt a parameter $ \theta_{j} $ $$ \frac{1}{N}\cdot\sum_{i}^{N}\left \| \frac{\partial {F(x_{i},...
1
vote
1answer
77 views

What does Goodfellow mean by “generator conditional variance”?

In Goodfellow's Generative Adversarial Nets, it is mentioned that Our work backpropagates derivatives through generative processes by using the observation that $$\lim_{\sigma \rightarrow 0} \...
0
votes
0answers
17 views

How the values in word2vec embedding are created for each word

I was going through word2vec materials from Andrew Ng's course and below is what i understood. -> Step1 A matrix of shape embedding_size*number_of_unique_words is created and populated with random ...
0
votes
0answers
12 views

Derivative of a particular cross entropy loss function [duplicate]

Suppose I have $E=-y\ log(y')$ and $y'=softmax(Vs)$. Please refer to here. How to compute $\partial E/\partial V$? I proceed as follows: $\partial E/\partial V=\partial E/\partial y'\ \partial y'/\...
0
votes
1answer
34 views

Not understanding backpropagation correctly

So I'm trying to build a simple NN where the layers are as follows: linear layer-> ReLu -> ...
0
votes
0answers
22 views

Literature / Example on Backpropagation Through Time Training of GRUs?

I am looking for a step by step explanation of BPTT Equations for GRUs. I found some for RNNs in general or LSTMs but none for GRUs. e.g. I am looking for something going into detail about Training ...
0
votes
1answer
82 views

How does Cross-Entropy (log loss) work with backpropagation?

I am having some trouble understanding how Cross Entropy would work with backpropagation. For backpropagation we exploit the chain rule to find the partial derivative of the Error function in terms of ...
0
votes
0answers
47 views

Why backpropagation if loss function is not convex in nature?

Backpropagation contains the method of gradient decent, which works well for convex loss functions with a global minima. But, for training, in most of the neural network tasks, backpropagation is ...
0
votes
1answer
34 views

Why Backpropagation is in negative

New to Backpropagation. In this example here and in all other example we calculate new weights by finding the derivative of the sigmoid https://repl.it/@vzhou842/An-Introduction-to-Neural-Networks ...
0
votes
0answers
46 views

Backprop through un-differentiable function f(x)?

Let's say I have a vector of gradients (ie. dL / dy). These gradients are the result of taking the derivative of the Loss function with respect to the output of a ...
0
votes
0answers
16 views

CNN Backpropagation Clarrification

Hi I am just trying to make sure my understanding of backpropagation with CNNs is correct, specifically CNNs that have multiple filters in each layer. This is how I have implemented backpropagation ...
0
votes
1answer
51 views

problem with vanishing/exploding gradient problems solution

I have few doubts around vanishing/exploding gradients. The problem with vanishing gradient is, When the weights are randomly initialized in a deep network, During back propagation initial layers ...
1
vote
1answer
65 views

Softmax in last layer - error rises but when using sigmoid error decreases [closed]

I wrote a neural network from scratch in Python. It has 1 hidden layer which uses tanh activation function. I train it on Iris and MNIST datasets. When I use Sigmoid in the last layer results are very ...
0
votes
0answers
20 views

pooling variable depth samplewise / are there any importantce to take into account?

My purpose is to apply pooling strategy (max/average) with a variable depth length of the samples in a batch. My question is whether is there something what I should take into account. Does it ...
0
votes
0answers
9 views

Backpropagation through an average gate

I'm currently going through the CS231n: CNN's for Computer Vision course offered for free by Stanford University and had a question regarding one part of the assignment. I'm currently trying to ...
0
votes
0answers
21 views

Backpropagation through 2D transposed convolution layer

I’m looking for an explanation for the backwards pass in a conv2d transpose layer. My main problem is that the deltas from the next layer are larger than the input of the previous layer. Hence, I can’...
0
votes
0answers
17 views

Computing derivatives for backpropagation across a convolution step

This will be a long post, but I hope it'll be instructive to anyone else in my position. I'm trying to find how the derivatives of the loss function are calculated with respect to the kernels and ...
0
votes
0answers
21 views

Correct gradient with custom weight update

I have a layer $f_{(a,b)}$, where $(a,b)$ are some parameters. During training, $(a,b)$ get updated using a custom update-scheme $g$. The thing is that $(a,b)$ don't get updated during the forward-...
0
votes
0answers
31 views

Faulty backpropagation of hidden layers? (C#)

I have been getting into machine learning and I decided to create my own NN Backpropagation program without libraries. I was making progress quite nicely for a while, but I got stuck and I can't seem ...
0
votes
0answers
12 views

Why cache gradients for params between training examples?

I was going through Karpathy's guide here where he defines an simple multiplication gate's forward and backward passes like so ...
0
votes
0answers
27 views

Difference between the convolution and correlation backpropagation

In the article about the convolution backpropagation, the computation of gradients to the input needs to rotate the weight and the computation of gradients to the weight also needs to rotate the input....
3
votes
1answer
130 views

How exactly is the error backpropagated in backpropagation?

I am reading a book on neural networks, and am now doing a chapter on backpropagation. (See chapter here). In this chapter, the writer is presenting four equations, that together form the backbone of ...
0
votes
1answer
30 views

2D max pool gradient propagation

I am trying to understand gradient propagation for a 2D max pooling operation when there is multiple filters for each position in the 2D grid (i.e. size = $b\times2\times2\times d$, where $b$ is ...
0
votes
0answers
35 views

Why doesn't my Feed-forward NN work when I try to train it with multiple inputs? [duplicate]

Basically I am trying to create a Neural Network in c# from scratch, without using any libraries. The issue that I am facing with right now is whenever I try to train my network with different inputs ...
0
votes
1answer
30 views

Binarizing Data in a Network using the sign function

I often see the use of the sign function in machine learning models as a way to binarize data (see eqn 1 here for an example). But the derivative of the sign function is the dirac delta function, so ...
0
votes
0answers
18 views

How to compute weight change for hidden layers with cross-entropy loss? [duplicate]

I'm trying to train a neural net with 1 hidden layer (RELU) softmax output layer cross-entropy loss stochastic gradient descent My implementation seems to work fine when I don't use any hidden ...
0
votes
0answers
32 views

What's the bug in my implementation/understanding of backpropagation?

For learning purposes, I'm trying to implement a simple neural network with only linear layers followed by logistic activation. As far as I understand, the backpropagation algorithm exploits the ...
0
votes
1answer
33 views

Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point: To calculate the gradients, you need to run the .backward method on a Variable, ...
0
votes
1answer
91 views

Softmax with Cross Entropy optimization vs Backpropagation

I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example). The code from the tutorial states "First we need to define the ...
4
votes
1answer
224 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
1
vote
1answer
19 views

Training Perceptrons with Backprop

Is it possible to train a simple perceptron with a threshold activation function such as this one: https://en.wikipedia.org/wiki/Perceptron with Backpropagation instead of the perceptron rule? is it ...
0
votes
1answer
25 views

Back propagation is done with each batch in a convolutional net, but is it also done with the validation set?

It's my understanding that the weights are updated in a convolutional neural network with each evaluation of a batch. But when the training data has been processed and it comes to predicting ...
0
votes
1answer
39 views

Confusion when Learning Parameters in BAYESIAN MODELS

I'm learning Bayesian Models but i still have some issues with the training of the parameters. These are my two questions : 1) Recall the Bayesian formula : $$p(\theta|X) = \frac{ p(X|\theta) \; p(\...
0
votes
3answers
66 views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
0
votes
1answer
27 views

Is backpropagation is used in validation data set? [closed]

Hello guys I am very confused as I am building a deep learning image classifier from raw python code ,so my question is that:-is backpropagation used in validation set to get the model more accuracy ...
0
votes
0answers
20 views

XOR Neural Network, Problem finding shapes of delta for backpropagation algorithm

I am taking the Machine Learning course by Andrew Ng on coursera. I am trying to make a neural network learn to do XOR, but I am facing a problem regarding the shapes of the $\delta$ vectors, and $\...
2
votes
1answer
36 views

Calculating the Policy Gradient for a Monte Carlo REINFORCE Algorithm

I am currently trying to implement the Monte Carlo REINFORCE algorithm, as described in Sutton and Barto's book Reinforcement Learning (p. 328, Second Edition). If $\theta$ denotes the parameter for ...
5
votes
2answers
324 views

Can $\sin(x)$ be used as activation in deep learning?

$\sin(x)$ seems to zero centered which is a desirable property for activation functions. Even the gradient won't vanish at any point. I am not sure if the oscillating nature of the function or its ...
0
votes
0answers
35 views

How to handle maxpool layer backpropagation with recurring max values in same position

Say I have a layer a: 3 4 2 1 5 0 8 6 4 The maxpool using 2x2 filter is: <...
0
votes
1answer
64 views

Meaning of “backpropagate through Gaussian distributions”?

I just started reading about GAN theory properly for the first time and I have a question about a comment in the original GAN paper. On page two there's a paragraph that states the following: ... ...
1
vote
1answer
14 views

How does backpropagation work with mixed architectures?

This question raised to me since I'm unsucessfully training a CNN-LSTM network atm. If for instance, LSTM requires a different type of BP algorithm (TBPTT), how do softwares deal with it? What is the ...
0
votes
0answers
16 views

Clarification on notation used to present back propagation algorithm in 'The Deep Learning Book'

In the deep learning book (free version is available online) the backpropation algorithm is explained in section 6.5. I have a question on equation (6.53): $$\frac{\partial u^{(n)}}{\partial u^{(j)}}...
0
votes
1answer
79 views

How is the loss(Backpropagation) for simple RNN calculated when dealing with batch?

I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself. However, it seems I have not ...