Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
96
votes
6answers
35k views

Is it possible to train a neural network without backpropagation?

Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient. Let's assume we are building a model with ~10K ...
43
votes
6answers
83k views

Backpropagation with Softmax / Cross Entropy

I'm trying to understand how backpropagation works for a softmax/cross-entropy output layer. The cross entropy error function is $$E(t,o)=-\sum_j t_j \log o_j$$ with $t$ and $o$ as the target and ...
22
votes
3answers
8k views

Why use gradient descent with neural networks?

When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to ...
19
votes
2answers
3k views

Backpropagation algorithm

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
6
votes
1answer
5k views

Neural network softmax activation

I'm trying to perform backpropagation on a neural network using Softmax activation on the output layer and a cross-entropy cost function. Here are the steps I take: Calculate the error gradient with ...
3
votes
1answer
142 views

Mean or sum of gradients for weight updates in SGD

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
27
votes
1answer
12k views

Why are non zero-centered activation functions a problem in backpropagation?

I read here the following: Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this soon) would be receiving ...
14
votes
3answers
2k views

Why back propagate through time in a RNN?

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs. Why would you not just ...
6
votes
2answers
709 views

My ReLU network fails to launch

So I have a problem. Simple situation: Fully-connected Multi-Layer Perceptron with Rectified Linear (ReLU) units (both hidden and output layers), 1 hidden layer of 100 hidden units, trained with ...
49
votes
1answer
70k views

How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
22
votes
1answer
10k views

Gradient backpropagation through ResNet skip connections

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer ...
20
votes
2answers
3k views

In neural nets, why use gradient methods rather than other metaheuristics?

In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent, Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics? By metaheuristics I mean ...
13
votes
2answers
4k views

Why can RNNs with LSTM units also suffer from “exploding gradients”?

I have a basic knowledge of how RNNs (and, in particular, with LSTMs units) work. I have a pictorial idea of the architecture of an LSTM unit, that is a cell and a few gates, which regulate the flow ...
6
votes
1answer
7k views

dropout: forward prop VS back prop in machine learning Neural Network

Regarding dropout, we know that in the forward propagation some neurons are put to "zero" (i.e., turned off). How about back propagation ? Are these dropped out neurons also zeros (turned off) ...
12
votes
2answers
3k views

Capturing initial patterns when using truncated backpropagation through time (RNN/LSTM)

Say that I use an RNN/LSTM to do sentiment analysis, which is a many-to-one approach (see this blog). The network is trained through a truncated backpropagation through time (BPTT), where the network ...
6
votes
1answer
1k views

How do Variational Auto Encoders backprop past the sampling step

From my understanding of VAE's, there's a step during training in the middle where, after the encoder produces a mean and standard deviation, random samples are drawn from the given learned ...
5
votes
1answer
565 views

Deep NNs, backpropagation and error calculation

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
0
votes
1answer
258 views

How are biases updated when 'batch size' > 1?

This is my network represented in matrices: (a dot represents an arbitrary number) Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity) Backpropagation The ...
37
votes
4answers
22k views

Why is tanh almost always better than sigmoid as an activation function?

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$. The reason he gives is that the outputs using $tanh$ ...
32
votes
7answers
28k views

Danger of setting all initial weights to zero in Backpropagation

Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?
34
votes
5answers
35k views

Backpropagation vs Genetic Algorithm for Neural Network training

I've read a few papers discussing pros and cons of each method, some arguing that GA doesn't give any improvement in finding the optimal solution while others show that it is more effective. It seems ...
13
votes
2answers
4k views

What is the benefit of the truncated normal distribution in initializing weights in a neural network?

When initializing connection weights in a feedforward neural network, it is important to initialize them randomly to avoid any symmetries that the learning algorithm would not be able to break. The ...
7
votes
3answers
11k views

What are the most popular artificial neural network algorithms for recognising the content of images?

What are the most used/popular artificial neural network algorithms for recognising the content of images in general? E.g. If the picture is of a person, dog, cat or a car. If the picture is a ...
15
votes
1answer
13k views

Sum or average of gradients in (mini) batch gradient decent?

When I implemented mini batch gradient decent, I just averaged the gradients of all examples in the training batch. However, I noticed that now the optimal learning rate is much higher than for online ...
20
votes
2answers
6k views

Why doesn't backpropagation work when you initialize the weights the same value?

Why doesn't backpropagation work when you initialize all the weight the same value (say 0.5), but works fine when given random numbers? Shouldn't the algorithm calculate the error and work from ...
11
votes
1answer
3k views

Training a convolution neural network

I am currently working on a face recognition software that uses convolution neural networks to recognize faces. Based on my readings, I've gathered that a convolutional neural network has shared ...
4
votes
1answer
54 views

What are the practical uses of Neural ODEs?

"Neural Ordinary Differential Equations", by Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud, was awarded the best-paper award in NeurIPS in 2018 There, authors propose the ...
4
votes
1answer
983 views

Why does SGD and back propagation work with ReLUs?

ReLUs are not differentiable at the origin. However, they are widely used in Deep Learning together with Stochastic Gradient Descent algorithms and Backpropagation, where the gradients of the loss ...
3
votes
1answer
270 views

Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
3
votes
1answer
167 views

Higher Order of Vectorization in Backpropagation in Neural Network

I am learning a machine learning class online from Stanford, namely CS 229. There is one section about deep learning and back-propagation in deep learning. The network looks like: The forward ...
3
votes
1answer
905 views

How to train a Recurrent Neural Network for Temporal XOR?

I have coded a Elman RNN using BackPropagation Through Time. In order to check my implementation, I have chosen Temporal XOR(a sequence of binary digits with the third being the xor of previous two ...
4
votes
1answer
2k views

Question with Matrix Derivative: Why do I have to transpose?

In the equation for Recurrent Neural Networks: $$ h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b) $$ Where $h_t$ is of size (N,H) Where $W_{hh}$ is of size (H,H) Where $W_{xh}$ is of size (D,H) Where $...
3
votes
4answers
2k views

Do extra hidden layers prevent convergence?

I have designed a simple feed-forward neural network using stochastic gradient descent. I use 22 inputs, 4 hidden layers, 1 output and am using a learning rate of 0.7 and momentum of 0.3. I have about ...
3
votes
0answers
94 views

Derive the gradients of a basic neural network

Given a neural network as following \begin{align*} &J = CE(y,\hat{y})=-\sum_i y_i log(\hat{y}_i)\\ &\hat{y} = softmax(z_2)\\ &z_2 = hW_2+b_2\\ &h = sigmoid(z_1)\\ &...
2
votes
1answer
879 views

Padding and stride in backpropagation of a conv net

I am trying to implement the back-propagation of a simple convolutional network. Specifically I understand that one of the steps is the convolution of the gradients coming from the next layer, with ...
2
votes
1answer
354 views

Deep NNs, backpropagation and error calculation [duplicate]

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
1
vote
1answer
4k views

Dropout backpropagation implementation

I understood the feedforward part of dropout during training, where for each example I multiply each activation with a binary mask to de-activate neurons with probability p. I use the inverted ...
0
votes
0answers
353 views

Back propagation cost function formula - what does $\sigma$ mean?

I'm a beginner in ML, and have a question on back propagation cost function. I'm reading the material on this link. In the below screen, since $a$ is the neuron's output, then why $a = \sigma(z)$ ...
0
votes
1answer
50 views

How are RNNs with inputs greater than the defined sequence length implamented

To clarify the slightly ambiguous language in the title. I have an RNN (actually 2 stacked RNN layers) that take input X of size X [batch_size, sequence_length, features] the model is trying to ...