# Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

39 questions
Filter by
Sorted by
Tagged with
35k views

### Is it possible to train a neural network without backpropagation?

Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient. Let's assume we are building a model with ~10K ...
83k views

### Backpropagation with Softmax / Cross Entropy

I'm trying to understand how backpropagation works for a softmax/cross-entropy output layer. The cross entropy error function is $$E(t,o)=-\sum_j t_j \log o_j$$ with $t$ and $o$ as the target and ...
8k views

### Why use gradient descent with neural networks?

When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to ...
3k views

### Backpropagation algorithm

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
5k views

### Neural network softmax activation

I'm trying to perform backpropagation on a neural network using Softmax activation on the output layer and a cross-entropy cost function. Here are the steps I take: Calculate the error gradient with ...
142 views

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
12k views

### Why are non zero-centered activation functions a problem in backpropagation?

I read here the following: Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this soon) would be receiving ...
2k views

### Why back propagate through time in a RNN?

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs. Why would you not just ...
709 views

### My ReLU network fails to launch

So I have a problem. Simple situation: Fully-connected Multi-Layer Perceptron with Rectified Linear (ReLU) units (both hidden and output layers), 1 hidden layer of 100 hidden units, trained with ...
70k views

### How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
10k views

### Gradient backpropagation through ResNet skip connections

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer ...
3k views

### In neural nets, why use gradient methods rather than other metaheuristics?

In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent, Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics? By metaheuristics I mean ...
4k views

### Why can RNNs with LSTM units also suffer from “exploding gradients”?

I have a basic knowledge of how RNNs (and, in particular, with LSTMs units) work. I have a pictorial idea of the architecture of an LSTM unit, that is a cell and a few gates, which regulate the flow ...
7k views

### dropout: forward prop VS back prop in machine learning Neural Network

Regarding dropout, we know that in the forward propagation some neurons are put to "zero" (i.e., turned off). How about back propagation ? Are these dropped out neurons also zeros (turned off) ...
3k views

### Capturing initial patterns when using truncated backpropagation through time (RNN/LSTM)

Say that I use an RNN/LSTM to do sentiment analysis, which is a many-to-one approach (see this blog). The network is trained through a truncated backpropagation through time (BPTT), where the network ...
1k views

### How do Variational Auto Encoders backprop past the sampling step

From my understanding of VAE's, there's a step during training in the middle where, after the encoder produces a mean and standard deviation, random samples are drawn from the given learned ...
565 views

### Deep NNs, backpropagation and error calculation

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
258 views

### How are biases updated when 'batch size' > 1?

This is my network represented in matrices: (a dot represents an arbitrary number) Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity) Backpropagation The ...
22k views

### Why is tanh almost always better than sigmoid as an activation function?

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$. The reason he gives is that the outputs using $tanh$ ...
28k views

### Danger of setting all initial weights to zero in Backpropagation

Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?
35k views

### Backpropagation vs Genetic Algorithm for Neural Network training

I've read a few papers discussing pros and cons of each method, some arguing that GA doesn't give any improvement in finding the optimal solution while others show that it is more effective. It seems ...
4k views

### What is the benefit of the truncated normal distribution in initializing weights in a neural network?

When initializing connection weights in a feedforward neural network, it is important to initialize them randomly to avoid any symmetries that the learning algorithm would not be able to break. The ...
11k views

### What are the most popular artificial neural network algorithms for recognising the content of images?

What are the most used/popular artificial neural network algorithms for recognising the content of images in general? E.g. If the picture is of a person, dog, cat or a car. If the picture is a ...
13k views

When I implemented mini batch gradient decent, I just averaged the gradients of all examples in the training batch. However, I noticed that now the optimal learning rate is much higher than for online ...
6k views

### Why doesn't backpropagation work when you initialize the weights the same value?

Why doesn't backpropagation work when you initialize all the weight the same value (say 0.5), but works fine when given random numbers? Shouldn't the algorithm calculate the error and work from ...
3k views

### Training a convolution neural network

I am currently working on a face recognition software that uses convolution neural networks to recognize faces. Based on my readings, I've gathered that a convolutional neural network has shared ...
54 views

### What are the practical uses of Neural ODEs?

"Neural Ordinary Differential Equations", by Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud, was awarded the best-paper award in NeurIPS in 2018 There, authors propose the ...
983 views

### Why does SGD and back propagation work with ReLUs?

ReLUs are not differentiable at the origin. However, they are widely used in Deep Learning together with Stochastic Gradient Descent algorithms and Backpropagation, where the gradients of the loss ...
270 views

### Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
167 views

### Higher Order of Vectorization in Backpropagation in Neural Network

I am learning a machine learning class online from Stanford, namely CS 229. There is one section about deep learning and back-propagation in deep learning. The network looks like: The forward ...
905 views

### How to train a Recurrent Neural Network for Temporal XOR?

I have coded a Elman RNN using BackPropagation Through Time. In order to check my implementation, I have chosen Temporal XOR(a sequence of binary digits with the third being the xor of previous two ...