# Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

341 questions
Filter by
Sorted by
Tagged with
1answer
1k views

### Backpropagation between pooling and convolutional layers

I've been working on understanding how convolutional neural networks by building my own implementation and trying to run a small network. So far I think I've gotten a good handle on the feed-forward ...
0answers
308 views

### Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:...
1answer
4k views

### Truncated Back-propagation through time for RNNs

I am not very clear on what is the proper way to train an RNN. Suppose we are using a vanilla RNN and are given some categorical sequence $x$ of length $T$: $$x= [ x_1,\ldots,x_T]$$ To fit the ...
1answer
6k views

### Backpropagation with Cross-entropy Cost Function

I'm using the cross-entropy cost function for backpropagation in a neutral network as it is discussed in neuralnetworksanddeeplearning.com. I got help on the cost function here: Cross-entropy cost ...
2answers
2k views

### Capturing initial patterns when using truncated backpropagation through time (RNN/LSTM)

Say that I use an RNN/LSTM to do sentiment analysis, which is a many-to-one approach (see this blog). The network is trained through a truncated backpropagation through time (BPTT), where the network ...
1answer
383 views

### Where can I find a clear derivation of backpropagation through a Convolutional Neural Network?

Any links to books, articles or papers would be appreciated, or even a written explanation.
1answer
791 views

### multidimensional inputs, outputs and backpropagation

Let's say I have a neural network in matrix form. Inputs, hidden layer nodes and outputs are represented by row vectors, while the weights are matrices of the sizes [outputRows; inputRows]. Now, let's ...
2answers
3k views

### In a neural network, do biases essentially need updates when being trained?

While building a neural network with one hidden layer, the question arose whether or not to update the biases during backpropagation. I'm basically trying to save up on memory, so my question was and ...
2answers
3k views

### How to train an SVM via backpropagation?

I was wondering if it was possible to train an SVM (say a linear one, to make things easy) using backpropagation? Currently, I'm at a road block, because I can only think about writing the classifier'...
0answers
204 views

### Having trouble understanding/implementing backpropagation algorithm

I have a simple feedforward neural network with 2 input neurons (and 1 bias neuron), 4 hidden neurons (and 1 bias neuron), and one output neuron. The feedforward mechanism seems to be working fine, ...
1answer
282 views

### What are the techniques used for learning in non-feedforward neural networks?

Suppose our network architecture has a hidden layer in which the hidden units are interconnected, then is there some sort of variation on backpropagation that is used? What about in general recurrent ...
1answer
818 views

### How to train a Recurrent Neural Network for Temporal XOR?

I have coded a Elman RNN using BackPropagation Through Time. In order to check my implementation, I have chosen Temporal XOR(a sequence of binary digits with the third being the xor of previous two ...
1answer
376 views

### Does Adding more neural units reduce the probability of trapping in a local minima?

Consider a multi-layer neural network that learns its weights with backpropagation (and gradient descent). Hence, there is a probability that we trap into a local minimum. Will adding more neural ...
0answers
132 views

### backpropagation: why find the global minimum instead of the value of zero

in back propagation, you use gradient descent to find the stationary point on the equation of the Error = (in terms of weight). But don't we want the error to be equal to zero? if the error is zero, ...
1answer
87 views

### Pros of back propagation learning algorithm

I'm doing some research into neural networks and it seems like every single one i've come across implements a back-propagation algorithm. Is this because they're very easy to implement or are there ...
2answers
1k views

### What is pretraining and how do you pretrain a neural network?

I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because ...
1answer
67k views

### How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
0answers
88 views

3answers
847 views

### Back propagation in Convolutional neural networks

I am trying to understand how CNN works. I want to use them in object recognition task. I thouhgt that CNN is unsupervised networks. My main question is how can I implement the back propagation ...
2answers
10k views

### Training a convolutional neural network

Based on my research on convolution neural networks, every other layer in such a network has a subsampling operation, in which the resolution of the image is reduced so as to improve generalization of ...
1answer
1k views

### How and why would MLPs for classification differ from MLPs for regression? Different backpropagation and transfer functions?

I'm using two 3-layer feedforward multi-layer perceptrons (MLPs). With the same input data (14 input neurons), I do one classification (true/false), and one regression (if true, "how much")¹. Until ...
2answers
1k views

### Backpropagation for convolutional network

I'm trying to develop the equations for back propagation for convolutional networks. Let's say I have a network with one convolutional network (no pooling) with no activation function. The activation ...
1answer
3k views

### Training a convolution neural network

I am currently working on a face recognition software that uses convolution neural networks to recognize faces. Based on my readings, I've gathered that a convolutional neural network has shared ...
1answer
4k views

### The tanh activation function in backpropagation

In the backpropagation algorithm when the output activation function is tanh and the number of classes is 2 (binary problem), the value obtained at the output layer ...
1answer
168 views

### How can I train in MLP backpropagation if training class labels are given with their confidence rate?

How can I make use of the information that shows confidence of that training instance? i.e. We have an extra information for training set about the confidence of class labels.
5answers
33k views

### Backpropagation vs Genetic Algorithm for Neural Network training

I've read a few papers discussing pros and cons of each method, some arguing that GA doesn't give any improvement in finding the optimal solution while others show that it is more effective. It seems ...
2answers
3k views

### Is ARIMA better in comparision with Neural Networks?

After working on Backpropagation Neural Network and ARIMA Time Series Model, I asked myself which one is better, but can't figure out the answer. They both use different approaches on the same problem ...
2answers
6k views

### Why doesn't backpropagation work when you initialize the weights the same value?

Why doesn't backpropagation work when you initialize all the weight the same value (say 0.5), but works fine when given random numbers? Shouldn't the algorithm calculate the error and work from ...
7answers
24k views

### Danger of setting all initial weights to zero in Backpropagation

Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?
2answers
2k views

### What sort of problems is backpropagation best suited to solving, and what are the best alternatives to backprop for solving those problems?

I am developing a neuroscience-inspired training algorithm for feed-forward neural networks. The natural benchmark for comparison is backpropagation. So I need to know to know what sort of ...
2answers
3k views

### Backpropagation algorithm

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...