Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
94
votes
6answers
33k views

Is it possible to train a neural network without backpropagation?

Many neural network books and tutorials spend a lot of time on the backpropagation algorithm, which is essentially a tool to compute the gradient. Let's assume we are building a model with ~10K ...
49
votes
1answer
68k views

How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
40
votes
6answers
75k views

Backpropagation with Softmax / Cross Entropy

I'm trying to understand how backpropagation works for a softmax/cross-entropy output layer. The cross entropy error function is $$E(t,o)=-\sum_j t_j \log o_j$$ with $t$ and $o$ as the target and ...
39
votes
1answer
21k views

How is softmax_cross_entropy_with_logits different from softmax_cross_entropy_with_logits_v2?

Specifically, I suppose I wonder about this statement: Future major versions of TensorFlow will allow gradients to flow into the labels input on backprop by default. Which is shown when I ...
34
votes
5answers
34k views

Backpropagation vs Genetic Algorithm for Neural Network training

I've read a few papers discussing pros and cons of each method, some arguing that GA doesn't give any improvement in finding the optimal solution while others show that it is more effective. It seems ...
31
votes
4answers
18k views

Why is tanh almost always better than sigmoid as an activation function?

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$. The reason he gives is that the outputs using $tanh$ ...
29
votes
7answers
25k views

Danger of setting all initial weights to zero in Backpropagation

Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?
26
votes
1answer
11k views

Why are non zero-centered activation functions a problem in backpropagation?

I read here the following: Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on this soon) would be receiving ...
22
votes
3answers
8k views

Why use gradient descent with neural networks?

When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to ...
22
votes
1answer
9k views

Gradient backpropagation through ResNet skip connections

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer ...
19
votes
2answers
6k views

Why doesn't backpropagation work when you initialize the weights the same value?

Why doesn't backpropagation work when you initialize all the weight the same value (say 0.5), but works fine when given random numbers? Shouldn't the algorithm calculate the error and work from ...
19
votes
2answers
3k views

In neural nets, why use gradient methods rather than other metaheuristics?

In training deep and shallow neural networks, why are gradient methods (e.g. gradient descent, Nesterov, Newton-Raphson) commonly used, as opposed to other metaheuristics? By metaheuristics I mean ...
17
votes
2answers
3k views

Backpropagation algorithm

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
14
votes
1answer
12k views

Sum or average of gradients in (mini) batch gradient decent?

When I implemented mini batch gradient decent, I just averaged the gradients of all examples in the training batch. However, I noticed that now the optimal learning rate is much higher than for online ...
14
votes
3answers
2k views

Why back propagate through time in a RNN?

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs. Why would you not just ...
13
votes
2answers
4k views

What is the benefit of the truncated normal distribution in initializing weights in a neural network?

When initializing connection weights in a feedforward neural network, it is important to initialize them randomly to avoid any symmetries that the learning algorithm would not be able to break. The ...
12
votes
2answers
3k views

Why can RNNs with LSTM units also suffer from “exploding gradients”?

I have a basic knowledge of how RNNs (and, in particular, with LSTMs units) work. I have a pictorial idea of the architecture of an LSTM unit, that is a cell and a few gates, which regulate the flow ...
12
votes
2answers
2k views

Capturing initial patterns when using truncated backpropagation through time (RNN/LSTM)

Say that I use an RNN/LSTM to do sentiment analysis, which is a many-to-one approach (see this blog). The network is trained through a truncated backpropagation through time (BPTT), where the network ...
12
votes
1answer
1k views

Matrix form of backpropagation with batch normalization

Batch normalization has been credited with substantial performance improvements in deep neural nets. Plenty of material on the internet shows how to implement it on an activation-by-activation basis. ...
11
votes
1answer
3k views

Training a convolution neural network

I am currently working on a face recognition software that uses convolution neural networks to recognize faces. Based on my readings, I've gathered that a convolutional neural network has shared ...
11
votes
2answers
10k views

How does minibatch gradient descent update the weights for each example in a batch?

If we process say 10 examples in a batch, I understand we can sum the loss for each example, but how does backpropagation work in regard to updating the weights for each example? For example: ...
11
votes
0answers
4k views

Backpropagation on a convolutional layer

Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually). First let me try ...
10
votes
2answers
3k views

How to train an SVM via backpropagation?

I was wondering if it was possible to train an SVM (say a linear one, to make things easy) using backpropagation? Currently, I'm at a road block, because I can only think about writing the classifier'...
10
votes
2answers
1k views

What is pretraining and how do you pretrain a neural network?

I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because ...
9
votes
1answer
3k views

Gradients for skipgram word2vec

I am going through the problems in the Stanford NLP deep learning class's written assignment problems http://cs224d.stanford.edu/assignment1/assignment1_soln I am trying to understand the answer for ...
8
votes
2answers
5k views

How to derive errors in neural network with the backpropagation algorithm?

From this video by Andrew Ng around 5:00 How are $\delta_3$ and $\delta_2$ derived? In fact, what does $\delta_3$ even mean? $\delta_4$ is got by comparing to y, no such comparison is possible for ...
8
votes
2answers
3k views

Recurrent Neural Network (RNN) topology: why always fully-connected?

I've started reading about Recurrent Neural Networks (RNNs) and Long Short Term Memory (LSTM) ...(...oh, not enough rep points here to list references...) One thing I don't get: It always seems that ...
8
votes
1answer
1k views

How and why would MLPs for classification differ from MLPs for regression? Different backpropagation and transfer functions?

I'm using two 3-layer feedforward multi-layer perceptrons (MLPs). With the same input data (14 input neurons), I do one classification (true/false), and one regression (if true, "how much")¹. Until ...
7
votes
3answers
10k views

What are the most popular artificial neural network algorithms for recognising the content of images?

What are the most used/popular artificial neural network algorithms for recognising the content of images in general? E.g. If the picture is of a person, dog, cat or a car. If the picture is a ...
7
votes
2answers
10k views

Training a convolutional neural network

Based on my research on convolution neural networks, every other layer in such a network has a subsampling operation, in which the resolution of the image is reduced so as to improve generalization of ...
7
votes
2answers
3k views

What is the problem with training Neural Networks with back propagation with activation functions that only output positive values? [duplicate]

I was watching the new CNNs course by Stanford (CS231n) and they mentioned in lecture 5 that its a bad idea to have activation functions that only output positive values. They explain the intuition on ...
7
votes
2answers
171 views

Clarification of the intuition behind backpropagation

I've been taking some time to try and understand the computations and mechanics of the machine learning algorithms I use in my day to day life. Studying the backpropagation literature on the CS231n ...
6
votes
1answer
648 views

How does backpropagation learn convolution filters?

I've understood how the backpropagation algorithm uses the partial derivatives of the weights to train a normal neural network. However, I cannot quite understand how the algorithm changes the filters....
6
votes
1answer
4k views

Neural network softmax activation

I'm trying to perform backpropagation on a neural network using Softmax activation on the output layer and a cross-entropy cost function. Here are the steps I take: Calculate the error gradient with ...
6
votes
3answers
6k views

How to find derivative of softmax function for the purpose of gradient descent?

I'm trying to understand back propagation algorithm for multiclass classification using gradient descent. I'm using https://www.cs.toronto.edu/~graves/phd.pdf . The output layer is a softmax layer, in ...
6
votes
2answers
3k views

Is ARIMA better in comparision with Neural Networks?

After working on Backpropagation Neural Network and ARIMA Time Series Model, I asked myself which one is better, but can't figure out the answer. They both use different approaches on the same problem ...
6
votes
2answers
2k views

Convolutional neural networks backpropagation

My question is regarding the answer to this question: Training a convolution neural network It seems like the answer is saying to change all the weights in a given filter by the same amount in the ...
6
votes
1answer
3k views

Why is Hebbian learning a less preferred option for training deep neural networks?

I understand that backpropagation is good, but what are the main advantages (and disadvantaged) that it has over Hebbian learning? I'm mostly wondering about contrastive Hebbian learning, though ...
6
votes
2answers
5k views

Backpropagation algorithm NN with Rectified Linear Unit (ReLU) activation

I am trying to follow a great example in R by Peng Zhao of a simple, "manually"-composed NN to classify the iris dataset into the three different species (setosa, ...
6
votes
1answer
553 views

Backpropagation algorithm in neural networks (NN) with logistic activation function

In this Coursera course by Geoffrey Hinton, the backpropagation algorithm is described starting at min 8 of this video, and when completed it looks like this: The slides can be found here. Now, the ...
6
votes
1answer
3k views

In RNN Back Propagation through time, why is the D(h_t)/D(h_(t-1)) diagonal?

I was going through backpropagation in time for RNN, in the deep learning book of Joshua Bengio et.al. (deep learning book , section 10.2.1 ). Given a network as: the book tells that the ...
5
votes
1answer
7k views

dropout: forward prop VS back prop in machine learning Neural Network

Regarding dropout, we know that in the forward propagation some neurons are put to "zero" (i.e., turned off). How about back propagation ? Are these dropped out neurons also zeros (turned off) ...
5
votes
2answers
228 views

Can $\sin(x)$ be used as activation in deep learning?

$\sin(x)$ seems to zero centered which is a desirable property for activation functions. Even the gradient won't vanish at any point. I am not sure if the oscillating nature of the function or its ...
5
votes
1answer
1k views

How do Variational Auto Encoders backprop past the sampling step

From my understanding of VAE's, there's a step during training in the middle where, after the encoder produces a mean and standard deviation, random samples are drawn from the given learned ...
5
votes
2answers
606 views

My ReLU network fails to launch

So I have a problem. Simple situation: Fully-connected Multi-Layer Perceptron with Rectified Linear (ReLU) units (both hidden and output layers), 1 hidden layer of 100 hidden units, trained with ...
5
votes
2answers
3k views

In a neural network, do biases essentially need updates when being trained?

While building a neural network with one hidden layer, the question arose whether or not to update the biases during backpropagation. I'm basically trying to save up on memory, so my question was and ...
5
votes
1answer
1k views

Transfer learning: How and why retrain only final layers of a network?

In this video, Prof. Andrew Ng says regarding transfer learning: Depending on how much data you have, you might just retrain the new layers of the network, or maybe you could retrain even more ...
5
votes
2answers
2k views

What sort of problems is backpropagation best suited to solving, and what are the best alternatives to backprop for solving those problems?

I am developing a neuroscience-inspired training algorithm for feed-forward neural networks. The natural benchmark for comparison is backpropagation. So I need to know to know what sort of ...
5
votes
1answer
179 views

What types of functions can be implemented in a layer of a Neural Networks?

One of the most common algorithms for training Neural Networks is back propagation, which essentially does (stochastic) gradient descent on the training objective function. Gradient descent can be ...
5
votes
1answer
1k views

Back-propagation in Convolution layer

Most examples I found on the internet explain well back-propagation in convolution layer, but only with a single kernel and single input channel. I do not understand how to do back-propagation for ...