Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

# Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

358 questions
Filter by
Sorted by
Tagged with
120 views

### Dimensions in single layer NN gradient

Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
517 views

### ReLU derivative - second order effects

I am reading the Deep Learning Book, where there is a section on generalisations of the ReLU (section 6.3.1). It states: The second derivative of the rectifying operation is 0 almost everywhere, ...
94 views

### Derive the gradients of a basic neural network

Given a neural network as following \begin{align*} &J = CE(y,\hat{y})=-\sum_i y_i log(\hat{y}_i)\\ &\hat{y} = softmax(z_2)\\ &z_2 = hW_2+b_2\\ &h = sigmoid(z_1)\\ &...
473 views

332 views

### Deep NNs, backpropagation and error calculation [duplicate]

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
702 views

### ReLU gradient descent matrix dimensionality

If I'm backpropagating through a recurrent neural network, say my layer output is $$h_t = \text{ReLU}(U h_{t-1} + V x_t),$$ when calculating the gradient my dimensions don't seem to be coming out ...
4k views

### Example of backpropagation for neural network with softmax and sigmoid activation

I am trying to produce a NN algorithm to classify the species of Iris into three species (versicolor, virginica, setosa) - preferably in R. The scaffolding / source is this code in R with ReLU ...
148 views

### Weights increasing in Multi-class NN

I am trying to make this network but here are the problems I find: In case of more than 1 hidden layer, I get nans as losses. In case of a single hidden layer, the loss first increase and then ...
1k views

### Explaining the distributions in Tensorflow's Tensorboard

I'm trying to train a cGAN network on Tensorflow and have all the summaries of the Discriminator, but I'm having difficulty understanding what they mean... There are currently 5 layers in the ...
2k views

### Feedforward Vs. Backpropagation Neural Network

I was taking the "Machine Learning-Coursera (Standford) by Andrew Ng" course. In Week 4 and Week 5 we have given programming ...
1k views

### Backpropagation: Is there a general weight update rule for both output and hidden layers?

I'm looking for a general weight update rule for both hidden and output layers, no matter the number of layers, the connections or the transfer function. Does anything like this exist? I'm quite new ...
1k views

### Backpropagation between pooling and convolutional layers

I've been working on understanding how convolutional neural networks by building my own implementation and trying to run a small network. So far I think I've gotten a good handle on the feed-forward ...
6k views

### Backpropagation with Cross-entropy Cost Function

I'm using the cross-entropy cost function for backpropagation in a neutral network as it is discussed in neuralnetworksanddeeplearning.com. I got help on the cost function here: Cross-entropy cost ...
35 views

### Calculating the Policy Gradient for a Monte Carlo REINFORCE Algorithm

I am currently trying to implement the Monte Carlo REINFORCE algorithm, as described in Sutton and Barto's book Reinforcement Learning (p. 328, Second Edition). If $\theta$ denotes the parameter for ...
85 views

### Backpropagation in a logarithmic layer of a regression NN

A "logarithmic neuron" is defined as follows : Which for inputs $\left\{ {{x_1},...,{x_n}} \right\}$ yields an output of $z=\prod\limits_{i = 1..n} {x_i^{{w_i}}}$ (in MATLAB, the activation ...
826 views

288 views

### Backpropagating a Dueling Architecture Network: Gradient Calculation

I started coding a Dueling Network Architectures for Deep Reinforcement Learning. I devided my network into two streams, arriving at a V(s) value and A(s,a) values. I arrived at the Q(s,a) output ...
727 views

### How was “derivative of the error function with respect to the activation” (that looks like y -t) derived?

In Chapter 5 (Neural Networks) of Bishop Pattern Recognition and Machine Learning he mentions several times that the derivative of the error function with respect to the activation for a particular ...
899 views

### Resilient Propagation: How to choose between RPROP+, RPROP-, iRPROP+, and iPROP-

I am using ENCOG to implement a Perceptron network. One of the easiest back-propagation (gradient descent) algorithms to use is the Resilient Propagation algorithm. There are four variants for ...
2k views

### Feed forward Neural Network and MSE issues

I've been implementing a Feed-forward Neural Network in C++ and CUDA. It is a basic Multi-layered Feed Forward ANN, using various activation functions (sigmoid bipolar, tanh, tanh scaled, and soft-...
820 views

### multidimensional inputs, outputs and backpropagation

Let's say I have a neural network in matrix form. Inputs, hidden layer nodes and outputs are represented by row vectors, while the weights are matrices of the sizes [outputRows; inputRows]. Now, let's ...
88 views

### Pros of back propagation learning algorithm

I'm doing some research into neural networks and it seems like every single one i've come across implements a back-propagation algorithm. Is this because they're very easy to implement or are there ...
479 views

### Why is backpropagation used more for fine-tuning than the up-down algorithm for deep belief networks?

Deep belief networks are pre-trained using RBMs then fine tuned for a supervised learning task. For almost every paper that I have read, I have seen back-propagation used instead of the up-down ...
171 views

### How can I train in MLP backpropagation if training class labels are given with their confidence rate?

How can I make use of the information that shows confidence of that training instance? i.e. We have an extra information for training set about the confidence of class labels.
42 views

### Looking for a linear optimization algorithm for the optimized shifting of time series

There is a set of time series (red) which get summed up to a cumulated time series (blue). During the optimization process, I want to scale the time series up or down so that my optimization ...
36 views

### How to compare the weight matrices from different gradient descent runs?

I would like to compare the end results of gradient descent for runs which use different initial weights and different orders of training data. I know that index by index in the matrix will make no ...
783 views

### Padding and stride in backpropagation of a conv net

I am trying to implement the back-propagation of a simple convolutional network. Specifically I understand that one of the steps is the convolution of the gradients coming from the next layer, with ...
100 views

### RNNs: backprop loss from just the last time step or every single one?

Consider a simple task that predict the next alphabets based on previous ones using RNNs. That is, during model inference, we would like the model to output y1_hat (...
57 views

### Working out the derivatives in backproagation

So I have no calculus experience what so ever and I've been tasked to build a neural network so finding the derivatives is proving quite problematic with my limited calculus experience. I've got the ...
349 views

### Why aren't vanishing gradients for deep networks a problem?

This wikipedia article for Autoencoders states the following [07.30am UTC, 28th November 2017]: An autoencoder is often trained using one of the many variants of backpropagation (such as conjugate ...
2k views

### Matrix Backpropagation with Softmax and Cross Entropy

I'm having trouble deriving the matrix form of backpropagation. As an example, let's suppose we have the following network: There are two nodes in the input layer plus a bias node fixed at 1, three ...
66 views

### How is a vector differentiation by a matrix defined?

I am a beginner of the machine learning. And at the same time, this is the first time to ask a question in this site, so I might have a problem with this question in terms of understandability. And ...
35 views

### Backprop working on output layer but not hidden layer [duplicate]

I wrote a program to classify MNIST with a vanilla neural net using sigmoid activation and back-propagation training. I tried to work through the math myself (because I want to understand things ), ...
42 views

### Neural Networks for predicting Energy at particular date

I am trying to predict Solar Energy value at particular date.So,for this I am applying Artificial Neural Networks model.I am having problem in deciding activation function. Since sigmoid function ...
328 views

### When doing backpropagation, why do we multiply with the derivative of the activation function

I was reading about backpropagation, which in my case works like this: I don't understand why when computing the error term, we are multiplying with the derivative of the activation function: Why ...
452 views

### Unsure if my implementation of a Convolutional layer doesn't learn or it's the correct behaviour

So for the last week or so I've tried to implement a Feed Forward network with multiple types of layers (Fully Connected, MaxPool, Convolution), multiple types of non-linear functions (tanh, sigmoid, ...