Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
2
votes
0answers
248 views

Simple Neural Network Issue in Python? [closed]

I thought it might be a good exercise to try my hand at making a simple, one hidden layer neural net from scratch. But, for whatever reason, I can't get my in-sample error to go down. I think it has ...
2
votes
0answers
89 views

Implementing backpropagation for dataset preprocessing

I have this algorithm written that can perform the XOR operation. How can i modify it so that it can process the large dataset? ...
2
votes
0answers
530 views

Scale MNIST-Data to [-0.9, 0.9]

I'm programming a neural network for MNIST-Recognition. My net has a pretty good performance, with accuracy > 98% on test set. But the training is very slow. So I thought it would be faster if I scale ...
2
votes
0answers
788 views

Fast RTRL(Real Time Recurrent Learning) for RNN

Assume generic RNN has update formula: $\mathbf{h}_{t+1} = f(\mathbf{x_t},\mathbf{h_t},\mathbf{\theta})$ [1] Where $\mathbf{x}$ is input vector, $\mathbf{h}$ is hidden state vector, and $\theta$ is ...
2
votes
1answer
395 views

Does Adding more neural units reduce the probability of trapping in a local minima?

Consider a multi-layer neural network that learns its weights with backpropagation (and gradient descent). Hence, there is a probability that we trap into a local minimum. Will adding more neural ...
2
votes
0answers
107 views

Training a MLP after pretraining RBMs with dropout

Let's say I have a couple of RBMs that I pretrained and that I used dropout. When finetuning, how does having used dropout effect backpropagation? Do I still use dropout while backprogating and change ...
1
vote
1answer
2k views

resilient backpropagation parameters selection

In the original paper for resilient backpropagation (http://paginas.fe.up.pt/~ee02162/dissertacao/RPROP%20paper.pdf), the author says "One of the main advantages of RPROP lies in the fact, that for ...
1
vote
2answers
373 views

Why is the second derivative required for newton's method for back-propagation?

I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely. I have seen this same question, and the widely accepted answer ...
1
vote
2answers
256 views

Backpropagation gradients don't match approximated gradients

I am in the process of implementing back propagation into my image classification neural net. I am using this cost function with a sigmoid output layer and ReLU hidden layers. The neural net has 3 ...
1
vote
1answer
4k views

Dropout backpropagation implementation

I understood the feedforward part of dropout during training, where for each example I multiply each activation with a binary mask to de-activate neurons with probability p. I use the inverted ...
1
vote
1answer
234 views

Derivative of the loss function w.r.t to X for the backpropagation

I would like to ask you why do we need to calculate a derivative of the loss function w.r.t X? It seems like, that for the backpropagation we need to calculate only a derivative w.r.t W. Can you ...
1
vote
1answer
440 views

How does backpropagation work in the case of reinforcement learning for games?

If we want a neural network to learn how to recognize e.g. digits, the backpropagation procedure is as follows: Let the NN look at an image of a digit, and output its probabilities on the different ...
1
vote
1answer
148 views

In Nielsen's explanation of backpropagation, why does the way he defines error change? Is it actually a change?

Specifically, why does Equation (BP1) not have the same form as (29)?1 In order to explain backpropagating the gradient through the neural net, he starts by defining something he calls error (...
1
vote
1answer
733 views

Implementing backpropagation in Theano

I am new to the machine learning field, so I am not sure if I am asking a dumb question. I have been playing around with Theano for a while and read a lot of code examples and it looks like every time ...
1
vote
1answer
70 views

What does Goodfellow mean by “generator conditional variance”?

In Goodfellow's Generative Adversarial Nets, it is mentioned that Our work backpropagates derivatives through generative processes by using the observation that $$\lim_{\sigma \rightarrow 0} \...
1
vote
1answer
45 views

Validate implementation of back-propagation algorithm

Let's say I implemented a CNN. Is there an easy way I can validate, that my implementation of back-propagation does not contain errors ? May be I can feed some dummy values into my network so it can ...
1
vote
1answer
195 views

Can all neural network cost functions be written as an average of individual cost and as a function of the activations at the output?

In chapter 2 of Michael Nielsen's Neural Networks and Deep Learning it says backpropagation relies on The first assumption we need is that the cost function can be written as an average $C = \frac{...
1
vote
1answer
57 views

What does $\mathcal{L}$ stands for in back propagation?

I am trying to learn how to do deep neural networks with this Ipython notebook. I'm puzzled about notations in linear backward learning section. For layer $l$, the linear part is: $Z^{[l]} = W^{[l]} ...
1
vote
1answer
219 views

Why don't neural networks get stuck in loops when they overshoot a backprop step?

In a normal feedforward network I wrote with linear activations I've noticed that after a while when the network has found a pretty viable solution to a problem it sometimes takes a step in the wrong ...
1
vote
1answer
61 views

The nature of the problem of vanishing gradients in RNN

In the context of RNNs, gradient vanishing refers to the fact the gradient signal decays to zero as we approach the beginning of the sequence during the unfolding of the network in backpropagation ...
1
vote
1answer
629 views

How does backpropagation differ from reverse-mode autodiff

Going through this book, I am familiar with the following: For each training instance the backpropagation algorithm first makes a prediction (forward pass), measures the error, then goes through ...
1
vote
1answer
345 views

Backpropagation in multi-layer perceptron (MLP) doesn't converge [closed]

My simple fully-connected multilayer perceptron (MLP) that I'm writing for academic purposes is causing to me sleep deprivation. I can't figure out why my MLP learns poorly, even if I try to solve ...
1
vote
1answer
1k views

Neural Network - Success after changing weights initialization strategy. What's the explanation?

I'm implementing a neural network in javascript to recognize handwritten digits, while studying "Neural Networks and Deep Learning" by Michael Nielsen and following the feedforward-backpropagation ...
1
vote
1answer
428 views

Explain this backpropagation graph?

Can someone help explain this backpropagation graph? This is from cs231n Convolutional Neural Networks for Visual Recognition by Stanford. So for this graph, let's say the true value of the output ...
1
vote
1answer
308 views

FeedForward Alternative to Backpropagation

I am reading a blog post that tries to explain backpropagation. In the build up the author shows how a naive method for computing gradients is sub-optimal. Consider this: Naive feedforward ...
1
vote
1answer
226 views

why should we transfer final state to initial state (BPTT) in LSTM?

I am learning LSTM implementation in torch from this code,it has these two lines of code: ...
1
vote
1answer
247 views

How to make sense of and use error derivatives of backpropagation algorithm

I'm following Geoffrey Hinton's Neural Networks Lectures (this one in particular: https://www.youtube.com/watch?v=LOc_y67AzCA), and I'm stumped by the second equation here: As I understand it, $E = \...
1
vote
1answer
48 views

How is the learning process for a NN implemented?

Thanks to BP and some scoring/loss functions, a NN can train to do the job as demanded. Or at least try to do so. I think I've understood BP but still I wonder about the following: Let's say there is ...
1
vote
1answer
61 views

Softmax in last layer - error rises but when using sigmoid error decreases [closed]

I wrote a neural network from scratch in Python. It has 1 hidden layer which uses tanh activation function. I train it on Iris and MNIST datasets. When I use Sigmoid in the last layer results are very ...
1
vote
1answer
19 views

Training Perceptrons with Backprop

Is it possible to train a simple perceptron with a threshold activation function such as this one: https://en.wikipedia.org/wiki/Perceptron with Backpropagation instead of the perceptron rule? is it ...
1
vote
1answer
14 views

How does backpropagation work with mixed architectures?

This question raised to me since I'm unsucessfully training a CNN-LSTM network atm. If for instance, LSTM requires a different type of BP algorithm (TBPTT), how do softwares deal with it? What is the ...
1
vote
1answer
29 views

Storage and re-computation of Intermediate / Weight / Back-propagated Gradients

I need to track the computation and storage of different parts of my network training. To be on the same page, let's assume the simple following scenario (biases omitted) Questions Local Gradients - ...
1
vote
1answer
87 views

Using step function as activation function in the final layer

I am using variational autoencoders as machine learning algorithm. My input data are images/matrices that represent user interface layouts or how the HTML page will be divided. I am thinking to ...
1
vote
1answer
1k views

Does dropout regularization prevent overfitting due to too many iterations?

For image classification problem, let's say, and given a neural network to train on, if you were to run too many iterations for a single image of a cat would not generalize well into other images of ...
1
vote
1answer
32 views

What does “learn the linear part of a mapping” mean?

In the paper "Efficient BackProp" , the authors talk about initializing the weights not too small and not too large: Intermediate weights that range over the sigmoid's linear region have the ...
1
vote
1answer
333 views

Why can't we use backpropagation and gradient descent on a Restricted Boltzmann Machine

Can someone please explain why we cannot use the backpropagation algorithm and gradient descent to train a Restricted Boltzmann Machine. In other words, why can't we train an RBM in the same manner ...
1
vote
1answer
31 views

Why caring about the dimensions and the axis of computation when calculating the derivative of the bias vector in backpropagation?

In order to create a linear backward function for my first deep neural network I wanted to calculate the derivative of $b$, the bias vector in the lth layer, $db^{[l]} = \frac{\partial \mathcal{L} }{\...
1
vote
1answer
56 views

Questions about Neural Network training (back propagation) in the book PRML (Pattern Recognition and Machine Learning)

I am reading Chapter 5 of PRML. Some symbols don't seem to be clear to me. In page 243, for the chain rule for partial derivative $\dfrac{\partial E_n}{\partial w_{ji}}=\dfrac{\partial E_n}{\partial ...
1
vote
1answer
393 views

how to derive the gradient of batch normalization

I'm trying to figure out the gradient of batch norm wrt x for backprop, but I get stuck in what I will call 'the triangle of (gradient) death'. I present to you the triangle of death (in red), in the ...
1
vote
1answer
60 views

NN to fill in blanks for desired output

Assume we have 1000 features that are fed into a NN classifier and that the NN is already trained well. The 1 output neuron has an activation > 0 to indicate the class ...
1
vote
1answer
618 views

SGD learning in CNN gets stuck when using a max pooling layer (x-post from DataScience) [closed]

I'm working on a CNN library for a university project and I'm having some trouble implementing the backpropagation through the max pooling layer. Please note that the whole thing was built from ...
1
vote
1answer
490 views

ANN: Weights grow very large, can I scale them?

I have written an ANN algorithm. And after several iterations my weights grow largely and there's this error which says the value of them is overflowing and therefore the outputs are NaNs. Does it ...
1
vote
1answer
526 views

What is the difference in “weight update process” in gradient descent vs Stochastic gradient descent?

Question In normal GD the weights are updated for every row in the training dataset while in SGD the weights are updated only once for the mini batch based on cummulative dLoss/dw1, dLoss/dw2 . Is my ...
1
vote
1answer
52 views

Are weights updated differently in a regression network vs. a classification network?

Are the weight of a neural network updated differently due to back propagation for a classification network vs. a regression network, if so how?.. My concern comes due to the both network uses ...
1
vote
2answers
702 views

Andrew Ng BackPropagation calculating partial derivative for the output layer [closed]

Given this equation -: ∆(l) = ∆(l) + δ(l+1)(a(l))T How can we calculate ∆(L) as this would require calculating δ(L+1) which never exists in the first place. I am stuck with Back Propagation ...
1
vote
1answer
54 views

What neural learning algorithm could work better than backpropagation for the dataset?

I'm writing a thesis where I developed a script that generates NN and precalculates weights and biases to reduce a required number of epochs. I am using feedforward and recurrent NN, applying ...
1
vote
1answer
845 views

Is it really necessary to flip the kernel in a conv-net?

If I have a software implemented convolutional network and the weights are trained via back propagation, is it really necessary to "flip" the kernel? Afterall I am applying the "unflipped" weights (eg ...
1
vote
1answer
411 views

How were neural network trained before backpropagation was proposed?

When learning neural networks one can often hear that Hinton proposed backpropagation in 1986. After this big leap forward, we could train neural network efficiently. But I have a question: How did ...
1
vote
1answer
23 views

Suspicion about the errorback propagation formula in PRML

From PRML: Do variations in $a_j$ give rise to variations in $E$ only via $a_k$? True. Is $h'(a_j)$ the same across all $k$? I'm afraid not. Let's write down what $\partial a_k/\partial a_j$ are....
1
vote
1answer
384 views

Why can't backpropagation be used for binary threshold neurons

I was reading that backpropagation can't be used for binary threshold neurons. Binary threshold neurons calculate a cost using some weights, so why can't those weights be changed with backpropagation?...