Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
3
votes
0answers
120 views

Dimensions in single layer NN gradient

Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
3
votes
0answers
517 views

ReLU derivative - second order effects

I am reading the Deep Learning Book, where there is a section on generalisations of the ReLU (section 6.3.1). It states: The second derivative of the rectifying operation is 0 almost everywhere, ...
3
votes
0answers
94 views

Derive the gradients of a basic neural network

Given a neural network as following \begin{align*} &J = CE(y,\hat{y})=-\sum_i y_i log(\hat{y}_i)\\ &\hat{y} = softmax(z_2)\\ &z_2 = hW_2+b_2\\ &h = sigmoid(z_1)\\ &...
2
votes
2answers
473 views

Can we use backpropagation to fit other models?

It appears that backpropagation is exclusively used to train neural network models. Why not use it to fit other models. For example - Taylor polynomials: $$ f(x) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)...
2
votes
3answers
994 views

Backward propagation without a loop over training examples

At this point in a Coursera Machine Learning lecture on implementing neural networks, Andrew Ng says: To implement back-prop, usually we will do that with a for-...
2
votes
1answer
4k views

The tanh activation function in backpropagation

In the backpropagation algorithm when the output activation function is tanh and the number of classes is 2 (binary problem), the value obtained at the output layer ...
2
votes
2answers
189 views

How does training affect the norm of weight matrices?

I have a neural network $F(W,x): \mathbb{R}^d \rightarrow \mathbb{R}^k$ with $L$ layers, $m$ neurons per layer, ReLu activation, softmax on the last layer and $n$ datapoint. My loss function is the ...
2
votes
1answer
5k views

Mean Absolute Error (MAE) derivative

$MAE=|y_{pred} - y_{true}|$ $\dfrac{dMAE}{dy_{pred}} = ?$ I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some ...
2
votes
1answer
2k views

Meaning of batch sizes for RNNs/LSTMs and reasons for padding

I've got a two conceptual questions about RNNs, particularly LSTMs, which I just can't figure out on my own or with the tutorials I find on the internet. I would really appreciate if you could help me ...
2
votes
2answers
520 views

I cannot differentiate my loss function, what is the best method for optimizing the weights in my neural network?

Suppose the output of my network is $y \in [0, 1]$ for a given input x. The loss function to be minimized is $f(x) = -\Sigma (w_i y_i + | y_i - y_{i-1}|)$, with real weights $w_i$. The dependency ...
2
votes
1answer
2k views

XOR backpropagation convergence

I've implemented 3 supervised training algorithms: rprop, online- and batch backprop with momentum. I have the simple XOR test, and I measured how many times they converge out of N iterations. My ...
2
votes
1answer
4k views

Cross Entropy Loss for One Hot Encoding

CE-loss sums up the loss over all output nodes $\sum_i[ - target_i*\log(output_i) ]$. The derivative of CE-loss is: $- \frac{target_i}{output_i}$. Since for a target=0 the loss and derivative of ...
2
votes
1answer
49 views

Backpropagation wrong? Doesn't it update dependent variables in hidden layer

In a multi layer perceptron or feedforward neural network, isn't backpropagation updating weights of the middle layers that are dependent variables? So for a particular hidden layer, it calculates all ...
2
votes
1answer
121 views

Why convoloution neural net have to find filter values ?

I'm new to ML stuff and one of the thing that I don't understand about CNN, is that why CNN have to find the values of filter at convolution layer, why don' they use existing filters and only find the ...
2
votes
1answer
1k views

Working for Logistic regression partial derivatives

In Andrew Ng's Neural Networks and Deep Learning course on Coursera the logistic regression loss function for a single training example is given as: $$ \mathcal L(a,y) = - \Big(y\log a + (1 - y)\log (...
2
votes
1answer
332 views

Deep NNs, backpropagation and error calculation [duplicate]

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
2
votes
2answers
702 views

ReLU gradient descent matrix dimensionality

If I'm backpropagating through a recurrent neural network, say my layer output is $$h_t = \text{ReLU}(U h_{t-1} + V x_t),$$ when calculating the gradient my dimensions don't seem to be coming out ...
2
votes
1answer
4k views

Example of backpropagation for neural network with softmax and sigmoid activation

I am trying to produce a NN algorithm to classify the species of Iris into three species (versicolor, virginica, setosa) - preferably in R. The scaffolding / source is this code in R with ReLU ...
2
votes
1answer
148 views

Weights increasing in Multi-class NN

I am trying to make this network but here are the problems I find: In case of more than 1 hidden layer, I get nans as losses. In case of a single hidden layer, the loss first increase and then ...
2
votes
1answer
1k views

Explaining the distributions in Tensorflow's Tensorboard

I'm trying to train a cGAN network on Tensorflow and have all the summaries of the Discriminator, but I'm having difficulty understanding what they mean... There are currently 5 layers in the ...
2
votes
1answer
2k views

Feedforward Vs. Backpropagation Neural Network

I was taking the "Machine Learning-Coursera (Standford) by Andrew Ng" course. In Week 4 and Week 5 we have given programming ...
2
votes
3answers
1k views

Backpropagation: Is there a general weight update rule for both output and hidden layers?

I'm looking for a general weight update rule for both hidden and output layers, no matter the number of layers, the connections or the transfer function. Does anything like this exist? I'm quite new ...
2
votes
1answer
1k views

Backpropagation between pooling and convolutional layers

I've been working on understanding how convolutional neural networks by building my own implementation and trying to run a small network. So far I think I've gotten a good handle on the feed-forward ...
2
votes
1answer
6k views

Backpropagation with Cross-entropy Cost Function

I'm using the cross-entropy cost function for backpropagation in a neutral network as it is discussed in neuralnetworksanddeeplearning.com. I got help on the cost function here: Cross-entropy cost ...
2
votes
1answer
35 views

Calculating the Policy Gradient for a Monte Carlo REINFORCE Algorithm

I am currently trying to implement the Monte Carlo REINFORCE algorithm, as described in Sutton and Barto's book Reinforcement Learning (p. 328, Second Edition). If $\theta$ denotes the parameter for ...
2
votes
1answer
85 views

Backpropagation in a logarithmic layer of a regression NN

A "logarithmic neuron" is defined as follows [1]: Which for inputs $\left\{ {{x_1},...,{x_n}} \right\}$ yields an output of $z=\prod\limits_{i = 1..n} {x_i^{{w_i}}}$ (in MATLAB, the activation ...
2
votes
1answer
826 views

Confusion about backpropagation - Matrix dimensions

Following Andrew Ng's notation. Suppose I wanted to implement a 4 layer neural network with the following weights, $\Theta_1 \in\mathbb{M}_{5\times 4},\:\Theta_2 \in\mathbb{M}_{5\times 6},\:\Theta_3 ...
2
votes
1answer
2k views

Can someone please explain the truncated back propagation through time algorithm?

I am reading about RNNs and how to train them and I understood how back propagation works. I have the following model: $$ h_t=f(Ah_{t-1}+ B x_t),\\ \hat{y}_t=g(C h_t). $$ For a given sample $(x_1^T,...
2
votes
2answers
288 views

Backpropagating a Dueling Architecture Network: Gradient Calculation

I started coding a Dueling Network Architectures for Deep Reinforcement Learning. I devided my network into two streams, arriving at a V(s) value and A(s,a) values. I arrived at the Q(s,a) output ...
2
votes
1answer
727 views

How was “derivative of the error function with respect to the activation” (that looks like y -t) derived?

In Chapter 5 (Neural Networks) of Bishop Pattern Recognition and Machine Learning he mentions several times that the derivative of the error function with respect to the activation for a particular ...
2
votes
1answer
899 views

Resilient Propagation: How to choose between RPROP+, RPROP-, iRPROP+, and iPROP-

I am using ENCOG to implement a Perceptron network. One of the easiest back-propagation (gradient descent) algorithms to use is the Resilient Propagation algorithm. There are four variants for ...
2
votes
1answer
2k views

Feed forward Neural Network and MSE issues

I've been implementing a Feed-forward Neural Network in C++ and CUDA. It is a basic Multi-layered Feed Forward ANN, using various activation functions (sigmoid bipolar, tanh, tanh scaled, and soft-...
2
votes
1answer
820 views

multidimensional inputs, outputs and backpropagation

Let's say I have a neural network in matrix form. Inputs, hidden layer nodes and outputs are represented by row vectors, while the weights are matrices of the sizes [outputRows; inputRows]. Now, let's ...
2
votes
1answer
88 views

Pros of back propagation learning algorithm

I'm doing some research into neural networks and it seems like every single one i've come across implements a back-propagation algorithm. Is this because they're very easy to implement or are there ...
2
votes
1answer
479 views

Why is backpropagation used more for fine-tuning than the up-down algorithm for deep belief networks?

Deep belief networks are pre-trained using RBMs then fine tuned for a supervised learning task. For almost every paper that I have read, I have seen back-propagation used instead of the up-down ...
2
votes
1answer
171 views

How can I train in MLP backpropagation if training class labels are given with their confidence rate?

How can I make use of the information that shows confidence of that training instance? i.e. We have an extra information for training set about the confidence of class labels.
2
votes
0answers
42 views

Looking for a linear optimization algorithm for the optimized shifting of time series

There is a set of time series (red) which get summed up to a cumulated time series (blue). During the optimization process, I want to scale the time series up or down so that my optimization ...
2
votes
0answers
36 views

How to compare the weight matrices from different gradient descent runs?

I would like to compare the end results of gradient descent for runs which use different initial weights and different orders of training data. I know that index by index in the matrix will make no ...
2
votes
1answer
783 views

Padding and stride in backpropagation of a conv net

I am trying to implement the back-propagation of a simple convolutional network. Specifically I understand that one of the steps is the convolution of the gradients coming from the next layer, with ...
2
votes
1answer
100 views

RNNs: backprop loss from just the last time step or every single one?

Consider a simple task that predict the next alphabets based on previous ones using RNNs. That is, during model inference, we would like the model to output y1_hat (...
2
votes
0answers
57 views

Working out the derivatives in backproagation

So I have no calculus experience what so ever and I've been tasked to build a neural network so finding the derivatives is proving quite problematic with my limited calculus experience. I've got the ...
2
votes
0answers
349 views

Why aren't vanishing gradients for deep networks a problem?

This wikipedia article for Autoencoders states the following [07.30am UTC, 28th November 2017]: An autoencoder is often trained using one of the many variants of backpropagation (such as conjugate ...
2
votes
0answers
2k views

Matrix Backpropagation with Softmax and Cross Entropy

I'm having trouble deriving the matrix form of backpropagation. As an example, let's suppose we have the following network: There are two nodes in the input layer plus a bias node fixed at 1, three ...
2
votes
0answers
66 views

How is a vector differentiation by a matrix defined?

I am a beginner of the machine learning. And at the same time, this is the first time to ask a question in this site, so I might have a problem with this question in terms of understandability. And ...
2
votes
0answers
35 views

Backprop working on output layer but not hidden layer [duplicate]

I wrote a program to classify MNIST with a vanilla neural net using sigmoid activation and back-propagation training. I tried to work through the math myself (because I want to understand things ), ...
2
votes
0answers
42 views

Neural Networks for predicting Energy at particular date

I am trying to predict Solar Energy value at particular date.So,for this I am applying Artificial Neural Networks model.I am having problem in deciding activation function. Since sigmoid function ...
2
votes
0answers
328 views

When doing backpropagation, why do we multiply with the derivative of the activation function

I was reading about backpropagation, which in my case works like this: I don't understand why when computing the error term, we are multiplying with the derivative of the activation function: Why ...
2
votes
1answer
452 views

Unsure if my implementation of a Convolutional layer doesn't learn or it's the correct behaviour

So for the last week or so I've tried to implement a Feed Forward network with multiple types of layers (Fully Connected, MaxPool, Convolution), multiple types of non-linear functions (tanh, sigmoid, ...
2
votes
1answer
243 views

CNN pooling and convolution

In the past, I've worked with the MNIST dataset and I am currently working with Java. I do not use any external libraries like numpy (for python) or something like that. I started from scratch and ...
2
votes
0answers
398 views

Deriving Gradients for a Vanilla LSTM

I've been banging my head on this for far too long. The following code should be easy to understand; can someone assist me in discovering what I've done wrong? The code passes a numerical gradient ...