Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
2
votes
1answer
1k views

Backpropagation between pooling and convolutional layers

I've been working on understanding how convolutional neural networks by building my own implementation and trying to run a small network. So far I think I've gotten a good handle on the feed-forward ...
5
votes
0answers
308 views

Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:...
5
votes
1answer
4k views

Truncated Back-propagation through time for RNNs

I am not very clear on what is the proper way to train an RNN. Suppose we are using a vanilla RNN and are given some categorical sequence $x$ of length $T$: $$x= [ x_1,\ldots,x_T]$$ To fit the ...
2
votes
1answer
6k views

Backpropagation with Cross-entropy Cost Function

I'm using the cross-entropy cost function for backpropagation in a neutral network as it is discussed in neuralnetworksanddeeplearning.com. I got help on the cost function here: Cross-entropy cost ...
11
votes
2answers
2k views

Capturing initial patterns when using truncated backpropagation through time (RNN/LSTM)

Say that I use an RNN/LSTM to do sentiment analysis, which is a many-to-one approach (see this blog). The network is trained through a truncated backpropagation through time (BPTT), where the network ...
1
vote
1answer
383 views

Where can I find a clear derivation of backpropagation through a Convolutional Neural Network?

Any links to books, articles or papers would be appreciated, or even a written explanation.
2
votes
1answer
791 views

multidimensional inputs, outputs and backpropagation

Let's say I have a neural network in matrix form. Inputs, hidden layer nodes and outputs are represented by row vectors, while the weights are matrices of the sizes [outputRows; inputRows]. Now, let's ...
5
votes
2answers
3k views

In a neural network, do biases essentially need updates when being trained?

While building a neural network with one hidden layer, the question arose whether or not to update the biases during backpropagation. I'm basically trying to save up on memory, so my question was and ...
10
votes
2answers
3k views

How to train an SVM via backpropagation?

I was wondering if it was possible to train an SVM (say a linear one, to make things easy) using backpropagation? Currently, I'm at a road block, because I can only think about writing the classifier'...
1
vote
0answers
204 views

Having trouble understanding/implementing backpropagation algorithm

I have a simple feedforward neural network with 2 input neurons (and 1 bias neuron), 4 hidden neurons (and 1 bias neuron), and one output neuron. The feedforward mechanism seems to be working fine, ...
1
vote
1answer
282 views

What are the techniques used for learning in non-feedforward neural networks?

Suppose our network architecture has a hidden layer in which the hidden units are interconnected, then is there some sort of variation on backpropagation that is used? What about in general recurrent ...
3
votes
1answer
818 views

How to train a Recurrent Neural Network for Temporal XOR?

I have coded a Elman RNN using BackPropagation Through Time. In order to check my implementation, I have chosen Temporal XOR(a sequence of binary digits with the third being the xor of previous two ...
2
votes
1answer
376 views

Does Adding more neural units reduce the probability of trapping in a local minima?

Consider a multi-layer neural network that learns its weights with backpropagation (and gradient descent). Hence, there is a probability that we trap into a local minimum. Will adding more neural ...
1
vote
0answers
132 views

backpropagation: why find the global minimum instead of the value of zero

in back propagation, you use gradient descent to find the stationary point on the equation of the Error = (in terms of weight). But don't we want the error to be equal to zero? if the error is zero, ...
2
votes
1answer
87 views

Pros of back propagation learning algorithm

I'm doing some research into neural networks and it seems like every single one i've come across implements a back-propagation algorithm. Is this because they're very easy to implement or are there ...
10
votes
2answers
1k views

What is pretraining and how do you pretrain a neural network?

I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because ...
49
votes
1answer
67k views

How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
1
vote
0answers
88 views

Regulation term disappears during taking gradient of cost function by using backpropagation

I'm trying to take the derivative of the cost function with respect to parameter $\theta$. The problem is $\frac{dJ(\theta)}{d\theta}$, somehow, is not equal to $\frac{dJ(\theta)}{dz}\cdot \frac{dz}{d\...
2
votes
0answers
106 views

Training a MLP after pretraining RBMs with dropout

Let's say I have a couple of RBMs that I pretrained and that I used dropout. When finetuning, how does having used dropout effect backpropagation? Do I still use dropout while backprogating and change ...
2
votes
1answer
475 views

Why is backpropagation used more for fine-tuning than the up-down algorithm for deep belief networks?

Deep belief networks are pre-trained using RBMs then fine tuned for a supervised learning task. For almost every paper that I have read, I have seen back-propagation used instead of the up-down ...
4
votes
2answers
4k views

backpropagation - bias nodes and error

I am implementing the stochastic gradient descent version of backpropagation from Tom Mitchell's Machine Learning book which has the steps for each training instance $\langle\vec{x},\vec{t}\rangle$: ...
2
votes
1answer
2k views

XOR backpropagation convergence

I've implemented 3 supervised training algorithms: rprop, online- and batch backprop with momentum. I have the simple XOR test, and I measured how many times they converge out of N iterations. My ...
3
votes
2answers
1k views

Full batch backpropagation implementation

I am trying to wrap my head around using batch backprop in a neural network. I have a very code-oriented mind, and I'm trying to figure out whether it's possible to parallelize the full batch ...
5
votes
1answer
632 views

How does backpropagation learn convolution filters?

I've understood how the backpropagation algorithm uses the partial derivatives of the weights to train a normal neural network. However, I cannot quite understand how the algorithm changes the filters....
3
votes
1answer
386 views

Is backpropagation algorithm same for both full-connected and local-connected neural network?

Is the backpropagation (BP) algorithm the same for both fully-connected and locally-connected (or partially-connected) neural networks? I know how to use BP for a fully-connected network, but I don't ...
7
votes
3answers
10k views

What are the most popular artificial neural network algorithms for recognising the content of images?

What are the most used/popular artificial neural network algorithms for recognising the content of images in general? E.g. If the picture is of a person, dog, cat or a car. If the picture is a ...
3
votes
2answers
2k views

Does a Neural Network actually need an activation function or is that just for Back Propagation?

I have a feed forward neural network (1 hidden layer with 10 neurons, 1 output layer with 1 neuron) with no activation function (only transfer by weight + bias) that can learn a really wonky sin wave (...
8
votes
2answers
4k views

How to derive errors in neural network with the backpropagation algorithm?

From this video by Andrew Ng around 5:00 How are $\delta_3$ and $\delta_2$ derived? In fact, what does $\delta_3$ even mean? $\delta_4$ is got by comparing to y, no such comparison is possible for ...
4
votes
3answers
847 views

Back propagation in Convolutional neural networks

I am trying to understand how CNN works. I want to use them in object recognition task. I thouhgt that CNN is unsupervised networks. My main question is how can I implement the back propagation ...
6
votes
2answers
10k views

Training a convolutional neural network

Based on my research on convolution neural networks, every other layer in such a network has a subsampling operation, in which the resolution of the image is reduced so as to improve generalization of ...
7
votes
1answer
1k views

How and why would MLPs for classification differ from MLPs for regression? Different backpropagation and transfer functions?

I'm using two 3-layer feedforward multi-layer perceptrons (MLPs). With the same input data (14 input neurons), I do one classification (true/false), and one regression (if true, "how much")¹. Until ...
1
vote
2answers
1k views

Backpropagation for convolutional network

I'm trying to develop the equations for back propagation for convolutional networks. Let's say I have a network with one convolutional network (no pooling) with no activation function. The activation ...
10
votes
1answer
3k views

Training a convolution neural network

I am currently working on a face recognition software that uses convolution neural networks to recognize faces. Based on my readings, I've gathered that a convolutional neural network has shared ...
2
votes
1answer
4k views

The tanh activation function in backpropagation

In the backpropagation algorithm when the output activation function is tanh and the number of classes is 2 (binary problem), the value obtained at the output layer ...
2
votes
1answer
168 views

How can I train in MLP backpropagation if training class labels are given with their confidence rate?

How can I make use of the information that shows confidence of that training instance? i.e. We have an extra information for training set about the confidence of class labels.
34
votes
5answers
33k views

Backpropagation vs Genetic Algorithm for Neural Network training

I've read a few papers discussing pros and cons of each method, some arguing that GA doesn't give any improvement in finding the optimal solution while others show that it is more effective. It seems ...
6
votes
2answers
3k views

Is ARIMA better in comparision with Neural Networks?

After working on Backpropagation Neural Network and ARIMA Time Series Model, I asked myself which one is better, but can't figure out the answer. They both use different approaches on the same problem ...
19
votes
2answers
6k views

Why doesn't backpropagation work when you initialize the weights the same value?

Why doesn't backpropagation work when you initialize all the weight the same value (say 0.5), but works fine when given random numbers? Shouldn't the algorithm calculate the error and work from ...
28
votes
7answers
24k views

Danger of setting all initial weights to zero in Backpropagation

Why is it dangerous to initialize weights with zeros? Is there any simple example that demonstrates it?
5
votes
2answers
2k views

What sort of problems is backpropagation best suited to solving, and what are the best alternatives to backprop for solving those problems?

I am developing a neuroscience-inspired training algorithm for feed-forward neural networks. The natural benchmark for comparison is backpropagation. So I need to know to know what sort of ...
17
votes
2answers
3k views

Backpropagation algorithm

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...