Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

Correct gradient with custom weight update

I have a layer $f_{(a,b)}$, where $(a,b)$ are some parameters. During training, $(a,b)$ get updated using a custom update-scheme $g$. The thing is that $(a,b)$ don't get updated during the forward-...
0
votes
0answers
19 views

Faulty backpropagation of hidden layers? (C#)

I have been getting into machine learning and I decided to create my own NN Backpropagation program without libraries. I was making progress quite nicely for a while, but I got stuck and I can't seem ...
0
votes
0answers
8 views

Why cache gradients for params between training examples?

I was going through Karpathy's guide here where he defines an simple multiplication gate's forward and backward passes like so ...
0
votes
0answers
11 views

Difference between the convolution and correlation backpropagation

In the article about the convolution backpropagation, the computation of gradients to the input needs to rotate the weight and the computation of gradients to the weight also needs to rotate the input....
3
votes
1answer
96 views

How exactly is the error backpropagated in backpropagation?

I am reading a book on neural networks, and am now doing a chapter on backpropagation. (See chapter here). In this chapter, the writer is presenting four equations, that together form the backbone of ...
0
votes
1answer
23 views

2D max pool gradient propagation

I am trying to understand gradient propagation for a 2D max pooling operation when there is multiple filters for each position in the 2D grid (i.e. size = $b\times2\times2\times d$, where $b$ is ...
0
votes
0answers
32 views

Why doesn't my Feed-forward NN work when I try to train it with multiple inputs? [duplicate]

Basically I am trying to create a Neural Network in c# from scratch, without using any libraries. The issue that I am facing with right now is whenever I try to train my network with different inputs ...
0
votes
1answer
20 views

Binarizing Data in a Network using the sign function

I often see the use of the sign function in machine learning models as a way to binarize data (see eqn 1 here for an example). But the derivative of the sign function is the dirac delta function, so ...
0
votes
0answers
15 views

How to compute weight change for hidden layers with cross-entropy loss? [duplicate]

I'm trying to train a neural net with 1 hidden layer (RELU) softmax output layer cross-entropy loss stochastic gradient descent My implementation seems to work fine when I don't use any hidden ...
0
votes
0answers
23 views

What's the bug in my implementation/understanding of backpropagation?

For learning purposes, I'm trying to implement a simple neural network with only linear layers followed by logistic activation. As far as I understand, the backpropagation algorithm exploits the ...
0
votes
1answer
28 views

Backpropagation gradient of the average

In the Pytorch Udacity course, the following is said at one point: To calculate the gradients, you need to run the .backward method on a Variable, ...
0
votes
1answer
26 views

Softmax with Cross Entropy optimization vs Backpropagation

I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example). The code from the tutorial states "First we need to define the ...
2
votes
1answer
54 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
1
vote
1answer
19 views

Training Perceptrons with Backprop

Is it possible to train a simple perceptron with a threshold activation function such as this one: https://en.wikipedia.org/wiki/Perceptron with Backpropagation instead of the perceptron rule? is it ...
0
votes
1answer
22 views

Back propagation is done with each batch in a convolutional net, but is it also done with the validation set?

It's my understanding that the weights are updated in a convolutional neural network with each evaluation of a batch. But when the training data has been processed and it comes to predicting ...
0
votes
1answer
34 views

Confusion when Learning Parameters in BAYESIAN MODELS

I'm learning Bayesian Models but i still have some issues with the training of the parameters. These are my two questions : 1) Recall the Bayesian formula : $$p(\theta|X) = \frac{ p(X|\theta) \; p(\...
0
votes
3answers
37 views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
0
votes
1answer
19 views

Is backpropagation is used in validation data set? [closed]

Hello guys I am very confused as I am building a deep learning image classifier from raw python code ,so my question is that:-is backpropagation used in validation set to get the model more accuracy ...
0
votes
0answers
16 views

XOR Neural Network, Problem finding shapes of delta for backpropagation algorithm

I am taking the Machine Learning course by Andrew Ng on coursera. I am trying to make a neural network learn to do XOR, but I am facing a problem regarding the shapes of the $\delta$ vectors, and $\...
2
votes
1answer
25 views

Calculating the Policy Gradient for a Monte Carlo REINFORCE Algorithm

I am currently trying to implement the Monte Carlo REINFORCE algorithm, as described in Sutton and Barto's book Reinforcement Learning (p. 328, Second Edition). If $\theta$ denotes the parameter for ...
5
votes
2answers
128 views

Can $\sin(x)$ be used as activation in deep learning?

$\sin(x)$ seems to zero centered which is a desirable property for activation functions. Even the gradient won't vanish at any point. I am not sure if the oscillating nature of the function or its ...
0
votes
0answers
28 views

How to handle maxpool layer backpropagation with recurring max values in same position

Say I have a layer a: 3 4 2 1 5 0 8 6 4 The maxpool using 2x2 filter is: <...
0
votes
1answer
32 views

Meaning of “backpropagate through Gaussian distributions”?

I just started reading about GAN theory properly for the first time and I have a question about a comment in the original GAN paper. On page two there's a paragraph that states the following: ... ...
1
vote
1answer
13 views

How does backpropagation work with mixed architectures?

This question raised to me since I'm unsucessfully training a CNN-LSTM network atm. If for instance, LSTM requires a different type of BP algorithm (TBPTT), how do softwares deal with it? What is the ...
0
votes
0answers
12 views

Clarification on notation used to present back propagation algorithm in 'The Deep Learning Book'

In the deep learning book (free version is available online) the backpropation algorithm is explained in section 6.5. I have a question on equation (6.53): $$\frac{\partial u^{(n)}}{\partial u^{(j)}}...
0
votes
1answer
67 views

How is the loss(Backpropagation) for simple RNN calculated when dealing with batch?

I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself. However, it seems I have not ...
0
votes
1answer
27 views

Backpropagation - Assumption or Error in Bishop's “PRaML”

I am currently reading Bishop - "Pattern Recognition and Machine Learning" (2006) and I could not figure out if I missed an assumption he made or if it is just wrong. In the chapter about ...
0
votes
0answers
118 views

backpropagation for multiple hidden layers

I'm currently implementing a small neural network library to learn the concepts coming with them better. As probably most people do, I'm struggling a bit with the backpropagation algorithm, especially ...
0
votes
0answers
30 views

Training convolutional neural network

I implemented Convolutional Neural Network from scratch for image recognition for 5 classes. When I train it using only one image from one class it seems to be working, because accuracy for this class ...
0
votes
0answers
39 views

Calculation of backpropagation of a NN with skip nodes

Assume this neural net. In our course we had to provide the forward and backward pass for this net. This is the solution we received: Even with the solutions provided I still have problems ...
0
votes
0answers
28 views

What are the advantage of using Sigmoid and Softmax and disadvantageous of both? [duplicate]

I am trying to understand the architecture of the neural network. I am supposed to make decision of how many hidden layers and what activation functions to use in the hidden layer and which activation ...
0
votes
0answers
15 views

ANN for Boundary Value Problem

I have a question regarding solving Boundary Value Problems (BVP) using ANNs. My understanding is that this is currently a challenging task. Most scientific literature on the subject is interested in ...
0
votes
2answers
178 views

Softmax layer derivative by hand

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=...
0
votes
1answer
87 views

Multiple filters during backpropagation in convolutional neural network

Let's say we have an Input 10x10x3 (WxHxD) and 5 filters 3x3x3. Convolution between Input and filters will be 8x8x5. During backpropagation we will get error with the same size 8x8x5. While ...
2
votes
2answers
107 views

How does training affect the norm of weight matrices?

I have a neural network $F(W,x): \mathbb{R}^d \rightarrow \mathbb{R}^k$ with $L$ layers, $m$ neurons per layer, ReLu activation, softmax on the last layer and $n$ datapoint. My loss function is the ...
1
vote
1answer
83 views

Derivative of the loss function w.r.t to X for the backpropagation

I would like to ask you why do we need to calculate a derivative of the loss function w.r.t X? It seems like, that for the backpropagation we need to calculate only a derivative w.r.t W. Can you ...
0
votes
0answers
34 views

Backpropagation through LSTM and MLP

For didactic reason, I am currently implementing in numpy an LSTM network for classifications. I need to add on top of the LSTM another fully connected layer, because I don't want the output to have ...
0
votes
1answer
31 views

Neural Networks - Back Propagation and Perceptrons

While studying about neural networks (still on basics - not Deep Learning etc.,) two questions came on my mind. What is the reason for replacing the hard limiter function in the nodes of the ...
0
votes
0answers
34 views

Mini Batch Gradient Descent Backpropapagation

I am a beginner to machine learning. I have derived the equations for backpropagation, and for the weight update for hidden layers, the update rule uses the output vector of the layer to multiply with ...
1
vote
0answers
21 views

Why is Backpropogation used instead of Rosenblatt's learning Algo or gradient descent to train MLP's?

In roesnblatt's learning algo and gradient descent the output is calculated for each input and based on the error b/w the outputs calculated and desired outputs the weights are updated. Why is ...
0
votes
0answers
22 views

Performing backpropagation on a 2 layer neural network

I am attempting to construct an NN from scratch without vectorizing. This is the network I'm attemping to model, where h1 and h2 represent the hidden layer nodes (h1 being the top, h2 being the ...
1
vote
0answers
63 views

Can you train deep recurrent neural network layer by layer?

Specifically for Gated Recurrent Unit, and say GRU is "layered" via but suppose it's only 2 layers deep for simplicity, and suppose the "total loss" = $L$ = $\sum{l_{t}} = \sum{error(y^{2}_{t})}$ ...
0
votes
1answer
33 views

How are RNNs with inputs greater than the defined sequence length implamented

To clarify the slightly ambiguous language in the title. I have an RNN (actually 2 stacked RNN layers) that take input X of size X [batch_size, sequence_length, features] the model is trying to ...
0
votes
0answers
28 views

Question about PyTorch tutorial

In this PyTorch tutorial the backprop to compute gradients is shown with the following code: ...
4
votes
3answers
117 views

how to make sense of the number of observations per parameters in deep learning models?

In a simple linear regression setting, it is common to talk about a minimum number of observations per parameter (which characterise the the degree of freedom). And it is easy to see that for multiple ...
2
votes
1answer
2k views

Cross Entropy Loss for One Hot Encoding

CE-loss sums up the loss over all output nodes $\sum_i[ - target_i*\log(output_i) ]$. The derivative of CE-loss is: $- \frac{target_i}{output_i}$. Since for a target=0 the loss and derivative of ...
0
votes
0answers
40 views

Backpropagation formula for simple ANN

I'm trying to solve an exercise. I'm asked to train a 2-layer neural network, given a small dataset. Below is a picture I drew to understand the architecture and derive the formula for backpropagation....
1
vote
1answer
25 views

Storage and re-computation of Intermediate / Weight / Back-propagated Gradients

I need to track the computation and storage of different parts of my network training. To be on the same page, let's assume the simple following scenario (biases omitted) Questions Local Gradients - ...
2
votes
1answer
45 views

Backpropagation wrong? Doesn't it update dependent variables in hidden layer

In a multi layer perceptron or feedforward neural network, isn't backpropagation updating weights of the middle layers that are dependent variables? So for a particular hidden layer, it calculates all ...
3
votes
2answers
2k views

MNIST digit recognition: what is the best we can get with a fully connected NN only? (no CNN)

To fully understand how it works internally, I'm re-writing a neural network from scratch in Python + numpy only. (As it's for learning purposes, performance is not an issue). Before moving to ...