Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
1
vote
1answer
2k views

Can someone please explain the truncated back propagation through time algorithm?

I am reading about RNNs and how to train them and I understood how back propagation works. I have the following model: $$ h_t=f(Ah_{t-1}+ B x_t),\\ \hat{y}_t=g(C h_t). $$ For a given sample $(x_1^T,...
1
vote
1answer
126 views

Matrix-based implementation for an ANN with different number of nodes per hidden layer?

Given a neural network architecture that has multiple hidden layers and a different number of nodes in each hidden layer (assuming that this is a valid architecture), is it possible to implement feed ...
1
vote
1answer
501 views

What is the update rule for hidden layer if softmax activation function is used?

I am trying to understand how backpropagation works. I understood the basic concepts and became familiar with derivation of equations for sigmoid activation function. Specifically for hidden layers, ...
1
vote
1answer
209 views

Intuition behind output neuron error signal in backpropagation

I have a potentially obvious question about neural network error backpropagation. In Andrej Karpathy's blog post on neural networks, he goes through an example of backpropagation. When calculating ...
1
vote
1answer
610 views

Concatenation of weights in Neural Networks

I'm training a special neural network for Natural Language Processing which has an embedding layer (this layer takes a one-hot vector of each word and output it's embedding vector through an embedding ...
1
vote
1answer
224 views

Vanishing gradient in basic 3-layer neural networks?

A 3-layer network has two layers of connections (between input and hidden layers and between hidden and output layer). Doesn't this mean that the gradient "vanishes", at least slighty, when training ...
1
vote
2answers
662 views

how to propagate error from convolutional layer to previous layer?

I've been trying to implement a simple convolutional neural network. But I stuck in some questions. To be specific, assume there are 3 layers in a convolutional pass, marked as ...
1
vote
1answer
310 views

Error Backpropagation, Christopher Bishop “Pattern Recognition and Machine Learning”

I'm trying to understand the description of the error backpropagation algorithm as explained in Christopher Bishop's book, in particular, section 5.3.1 "Evaluation of error-function derivative". The ...
1
vote
1answer
156 views

Back propagation error calculation

I'm working through AI: A Modern Approach by Russel and Norvig. At the section on back-propagation, they have this to say: The idea is that hidden node j is responsible for some fraction of the ...
1
vote
1answer
124 views

Drop Connect Back Propagation

I'm trying to implement drop connect. Am I supposed to use the same drop masks during back propagation?
1
vote
2answers
1k views

Backpropagation for convolutional network

I'm trying to develop the equations for back propagation for convolutional networks. Let's say I have a network with one convolutional network (no pooling) with no activation function. The activation ...
1
vote
0answers
33 views

Looking for a linear optimization algorithm for the optimized shifting of time series

There is a set of time series (red) which get summed up to a cumulated time series (blue). During the optimization process, I want to scale the time series up or down so that my optimization ...
1
vote
0answers
21 views

Why is Backpropogation used instead of Rosenblatt's learning Algo or gradient descent to train MLP's?

In roesnblatt's learning algo and gradient descent the output is calculated for each input and based on the error b/w the outputs calculated and desired outputs the weights are updated. Why is ...
1
vote
0answers
92 views

Can you train deep recurrent neural network layer by layer?

Specifically for Gated Recurrent Unit, and say GRU is "layered" via but suppose it's only 2 layers deep for simplicity, and suppose the "total loss" = $L$ = $\sum{l_{t}} = \sum{error(y^{2}_{t})}$ ...
1
vote
0answers
21 views

Batch learning in digits recognition (MNIST database) [duplicate]

While working my way through M. Nielsen's "Neural networks and deep learning", I decided to try out some presumably silly things to really understand why they won't work and/or why it's not a good ...
1
vote
0answers
392 views

Need help writing a neural network for a Pokemon battle

I'm trying to write a neural network that's able to select the optimal course of action in a Pokemon battle. In a battle, there are two different types of actions: use one of the four moves known by ...
1
vote
1answer
93 views

For back propagation in neural networks , how do we calculate vector by matrix derivative?

I am following the course deep learning ai by Andrew NG. In course1 week4, 04-06-Forward and Backward Propagation, he calculates backward propagation for layer $l$ in neural networks as follows (a ...
1
vote
1answer
363 views

Over which set of elements should I perform norm clipping of gradients for backpropagation?

I want to normalise the gradients of my multi-layer perceptron in order to avoid the Exploding Gradients Problem, so I thought I would use l2-normalisation but am unsure about how to apply it to the ...
1
vote
0answers
218 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
1
vote
1answer
43 views

backpropagation; differentiation w.r.t bias

Probable, it is a silly question, but I cannot handle it at the moment. Given: input: 3 hidden1: 4 hidden2: 4 output: 2 Output layer does not have any activation function. My question is, ...
1
vote
0answers
46 views

Can Regularization by achieved using Relative Sensitivity?

In a Mathematical Model we measure the sensitivity of the output with respect to the parameters and it is desirable that a small change in a parameter doesn't lead to wild fluctuations in the output ...
1
vote
0answers
911 views

Numerical gradient checking (best practices)

I've implemented a neural network and am using numerical gradient checking to validate the back-propagation algorithm is working correctly. I'm using the standard method to calculate the numerical ...
1
vote
0answers
367 views

finding the loss derivative w.r.t. weights for a convolutional layer

Take the loss function: $$ \mathrm{loss} ~~=~~ \sum_{i=1}^N \left( -z_{}[y] + \log{\left( \sum_{c=1}^{10} \exp(z_{}[c]) \right)} \right)$$ where $z \in \mathbb{R}^{10}$ is the input to the softmax ...
1
vote
1answer
225 views

Inception module backpropagation - how to get the input error?

I'm implementing an Inception module from scratch, specifically this version with dimensionality reduction, and I'm not sure about how to calculate the input error delta when doing the backpropagation ...
1
vote
0answers
133 views

Minibatch BPTT application in recurrent neural nets

I'm trying to wrap my head around the finer points of backpropagation through time and was hoping to get some clarification on something. For the most part, I understand the general idea of BPTT: we ...
1
vote
0answers
161 views

Is my Neural Network chain rule correct?

Suppose I have the following architecture Where $ HA_i$ and $OA_i$ are the activated values of the hidden and output nodes respectively, and $W_i$ are the weights between nodes. I want to find ...
1
vote
0answers
80 views

Interpretations of chain rule for backprop

In Goodfellow et al.'s Deep Learning, the authors write on page 203: Let $w \in \mathbb{R}$ be the input to the graph. We use the same function $f: \mathbb{R} \rightarrow \mathbb{R}$ as the ...
1
vote
1answer
53 views

Is it possible to premultiply a Neural Network during the forward pass?

A Neural Network is essentially a set of weight matrices. Let's call them H1, H2, H3 etc (where each index is the hidden layer number). In the forward pass, we take the input batch/dataset as a ...
1
vote
0answers
62 views

Is there any method for learning regularization parameter? [duplicate]

One of the hardest tasks in using machine learning methods is choosing the appropriate hyper-parameters of the model such as regularization parameter. As far as I know, this task is performed by a ...
1
vote
2answers
2k views

What is the meaning of the error rate in Neural networks?

I'm a beginner with neural networks. I get that the error should be low. But what does that number really mean? I created a simple neural network which has a error of 1.5. Is that too high? What are ...
1
vote
1answer
50 views

Replicator Neuron Network back propagation problem

Recently I started reading this paper: Link, and found it quite interesting, so I start to write the code Python. The code run with no error, however, the result is not like my expectation. After a ...
1
vote
0answers
36 views

Neural networks: Why is is neccesary to shuffle the inputs in each epoque? [duplicate]

Why is is neccesary to shuffle the inputs in each backpropagation of each epoque? What happens if they are not shuffle?
1
vote
1answer
242 views

Heterogenous layers in Neural Network

For this question when I say Neural Network I mean the standard Multi-layered Fully Connected Feed Forward Neural Network. I'm exploring the different activation functions and cost functions used in ...
1
vote
0answers
289 views

Recursive backpropagation vs backpropagation

I recently read a paper that tried to find out what happened in a RNN by linearization around slow and fixed points. I can't figure out why we have to use linearization around these points. After I ...
1
vote
0answers
33 views

does bp need to be simultanious? - XOR learning problem

I have a problem with XOR learning. I have 2(inputs),2,1(output) neuron neural network with sigmoidal function and normal sets ...
1
vote
0answers
70 views

Elaboration on weight change for output layer in neural network

I have trouble understanding the section regarding backpropagation in Murphy's Machine Learning book. He derives the weight change for the output layer (16.68) as follows: $\nabla_{\mathbf{w}_k} J_n ...
1
vote
0answers
733 views

Backpropagation for Bias in Neural Networks

I have a problem in my neural network relating to the bias vector. I'm using this source as a reference. My understanding of calculating the bias is that the partial derivative cost function with ...
1
vote
1answer
92 views

Under periodic BPTT, is softmax evaluated only at the end of the period?

Suppose I have a continuous sequence $X$ of words and I wish to train a RNN language model. According to [1], I would split $X$ into subsequences $X^{1..|X|/k_1}$ $k_1$ sized subsequences ($k_1$ is ...
1
vote
0answers
34 views

Neural Network probabilities converging to biases

I'm creating an Android app which can use a variety of classification formula, and while I have normal Softmax done correctly, I keep having an issue with the Softmax Neural Network. After about 10 ...
1
vote
0answers
80 views

how does a neural network with stochastic backpropagation make sure it doesn't “undo” previous learning?

Assume we have a neural network with stochastic gradient descent used for backpropagation, and therefore each element in the training set is used once to calculate the error, and then to adjust the ...
1
vote
1answer
197 views

Backpropagation Through Time Error Computation

I'm attempting to work through the backpropagation through time terms using this source: http://www.deeplearningbook.org/contents/rnn.html The final formulas are given on pages 385 and 386, but I ...
1
vote
1answer
856 views

Is it normal that a Neural Network sometimes doesn't learn Xor?

I've implemented a neural network and I'm training it to compute Xor. 1 out of x times it fails to learn, where x is about 5 or 10. It then gives e.g. 0.67 instead of 0 as output for input (1,1). Is ...
1
vote
0answers
146 views

Neural Networks — How to design for multiple outputs [duplicate]

Given a multilayer perceptron to be used to separate 2 classes. One has 2 design options. 1 -- Use one output node -- where one class is trained to give an output equal to zero, and the other class ...
1
vote
1answer
2k views

Simple Neural Network for time series prediction

I am creating a simple Multi-layered feed forward Neural Network using AForge.net NN library. My NN is a 3 Layered Activation Network trained with Supervised Learning approach using BackPropogation ...
1
vote
1answer
385 views

Where can I find a clear derivation of backpropagation through a Convolutional Neural Network?

Any links to books, articles or papers would be appreciated, or even a written explanation.
1
vote
0answers
205 views

Having trouble understanding/implementing backpropagation algorithm

I have a simple feedforward neural network with 2 input neurons (and 1 bias neuron), 4 hidden neurons (and 1 bias neuron), and one output neuron. The feedforward mechanism seems to be working fine, ...
1
vote
1answer
306 views

What are the techniques used for learning in non-feedforward neural networks?

Suppose our network architecture has a hidden layer in which the hidden units are interconnected, then is there some sort of variation on backpropagation that is used? What about in general recurrent ...
1
vote
0answers
139 views

backpropagation: why find the global minimum instead of the value of zero

in back propagation, you use gradient descent to find the stationary point on the equation of the Error = (in terms of weight). But don't we want the error to be equal to zero? if the error is zero, ...
1
vote
0answers
90 views

Regulation term disappears during taking gradient of cost function by using backpropagation

I'm trying to take the derivative of the cost function with respect to parameter $\theta$. The problem is $\frac{dJ(\theta)}{d\theta}$, somehow, is not equal to $\frac{dJ(\theta)}{dz}\cdot \frac{dz}{d\...
0
votes
1answer
111 views

What is the expression for derivative of the signum function one should use in the BP training method

The back propagation learning method requires knowing of derivatives of activation functions. But what expression one should use for signum activation function $$ \mathrm{Signum}( x ) = \...