Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
1
vote
1answer
23 views

Suspicion about the errorback propagation formula in PRML

From PRML: Do variations in $a_j$ give rise to variations in $E$ only via $a_k$? True. Is $h'(a_j)$ the same across all $k$? I'm afraid not. Let's write down what $\partial a_k/\partial a_j$ are....
1
vote
1answer
387 views

Why can't backpropagation be used for binary threshold neurons

I was reading that backpropagation can't be used for binary threshold neurons. Binary threshold neurons calculate a cost using some weights, so why can't those weights be changed with backpropagation?...
2
votes
0answers
332 views

When doing backpropagation, why do we multiply with the derivative of the activation function

I was reading about backpropagation, which in my case works like this: I don't understand why when computing the error term, we are multiplying with the derivative of the activation function: Why ...
3
votes
0answers
402 views

Batch-normalization back propagation equation

I have been trying to understand applying the back propagation on batch-normalization. However, the matrix multiplication and equations involved have got me confused and I feel that my understanding ...
2
votes
1answer
848 views

Confusion about backpropagation - Matrix dimensions

Following Andrew Ng's notation. Suppose I wanted to implement a 4 layer neural network with the following weights, $\Theta_1 \in\mathbb{M}_{5\times 4},\:\Theta_2 \in\mathbb{M}_{5\times 6},\:\Theta_3 ...
2
votes
2answers
723 views

ReLU gradient descent matrix dimensionality

If I'm backpropagating through a recurrent neural network, say my layer output is $$h_t = \text{ReLU}(U h_{t-1} + V x_t),$$ when calculating the gradient my dimensions don't seem to be coming out ...
1
vote
1answer
49 views

Is training a deep neural network still referred as training using back propagation?

I wa currently reading up on standard neural network and become a bit confused in the terms used relating training deep neural network versus a normal neural network. Are they trained similarly or ...
1
vote
2answers
2k views

What is the meaning of the error rate in Neural networks?

I'm a beginner with neural networks. I get that the error should be low. But what does that number really mean? I created a simple neural network which has a error of 1.5. Is that too high? What are ...
-1
votes
1answer
67 views

Training values for neural network

In order to help myself understand neural networks better, I'm attempting to write the code for a multilayered neural network in Python. I've written the code for predicting the output, given a set ...
2
votes
1answer
457 views

Unsure if my implementation of a Convolutional layer doesn't learn or it's the correct behaviour

So for the last week or so I've tried to implement a Feed Forward network with multiple types of layers (Fully Connected, MaxPool, Convolution), multiple types of non-linear functions (tanh, sigmoid, ...
1
vote
1answer
980 views

Back propagation and and various cost functions

I want to understand and extract a codeable back propagation algorithm. I'm mostly pure ignorant coder and my math skills are very weak. And that what I want to go further with is an extendable ...
3
votes
0answers
121 views

Dimensions in single layer NN gradient

Given a neural network with one hidden sigmoid layer and softmax output layer, I want to derive the gradient of the cross entropy loss with respect to the first weight matrix. This is equivalent to ...
6
votes
1answer
5k views

Neural network softmax activation

I'm trying to perform backpropagation on a neural network using Softmax activation on the output layer and a cross-entropy cost function. Here are the steps I take: Calculate the error gradient with ...
0
votes
1answer
546 views

Calculating Softmax derivative independent of cost function

Note: there is a nearly identical question on Stack Overflow. However, I seem to be missing something... or maybe it's just that Python isn't my first language ;) For a neural network library, I've ...
22
votes
1answer
10k views

Gradient backpropagation through ResNet skip connections

I'm curious about how gradients are back-propagated through a neural network using ResNet modules/skip connections. I've seen a couple of questions about ResNet (e.g. Neural network with skip-layer ...
2
votes
1answer
4k views

Example of backpropagation for neural network with softmax and sigmoid activation

I am trying to produce a NN algorithm to classify the species of Iris into three species (versicolor, virginica, setosa) - preferably in R. The scaffolding / source is this code in R with ReLU ...
6
votes
2answers
5k views

Backpropagation algorithm NN with Rectified Linear Unit (ReLU) activation

I am trying to follow a great example in R by Peng Zhao of a simple, "manually"-composed NN to classify the iris dataset into the three different species (setosa, ...
6
votes
1answer
614 views

Backpropagation algorithm in neural networks (NN) with logistic activation function

In this Coursera course by Geoffrey Hinton, the backpropagation algorithm is described starting at min 8 of this video, and when completed it looks like this: The slides can be found here. Now, the ...
11
votes
2answers
10k views

How does minibatch gradient descent update the weights for each example in a batch?

If we process say 10 examples in a batch, I understand we can sum the loss for each example, but how does backpropagation work in regard to updating the weights for each example? For example: ...
2
votes
1answer
148 views

Weights increasing in Multi-class NN

I am trying to make this network but here are the problems I find: In case of more than 1 hidden layer, I get nans as losses. In case of a single hidden layer, the loss first increase and then ...
2
votes
1answer
1k views

Explaining the distributions in Tensorflow's Tensorboard

I'm trying to train a cGAN network on Tensorflow and have all the summaries of the Discriminator, but I'm having difficulty understanding what they mean... There are currently 5 layers in the ...
3
votes
2answers
1k views

How to update kernel values in a convolutional layer during backward pass?

I started coding backpropagation for a simple convnet and had some troubles understanding the algorithm. I do get the idea of weight update based on gradients, but because the filter kernel parameters ...
1
vote
1answer
50 views

Replicator Neuron Network back propagation problem

Recently I started reading this paper: Link, and found it quite interesting, so I start to write the code Python. The code run with no error, however, the result is not like my expectation. After a ...
3
votes
0answers
528 views

ReLU derivative - second order effects

I am reading the Deep Learning Book, where there is a section on generalisations of the ReLU (section 6.3.1). It states: The second derivative of the rectifying operation is 0 almost everywhere, ...
3
votes
2answers
2k views

Matrix and Vector Approaches to Backpropagation in a Neural Network

I recently implemented a neural network, with backpropagation in a fully matrix approach, as described here, where the whole dataset is used for each backprop: http://ufldl.stanford.edu/wiki/index.php/...
0
votes
1answer
969 views

Training neural network on skewed dataset: output always 1

I'm learning neural networks, and wrote a network from scratch using numpy and pandas. I'm training it using stochastic gradient descent to predict house prices. The dataset is right-skewed, I mean-...
1
vote
0answers
38 views

Neural networks: Why is is neccesary to shuffle the inputs in each epoque? [duplicate]

Why is is neccesary to shuffle the inputs in each backpropagation of each epoque? What happens if they are not shuffle?
3
votes
2answers
545 views

Why does pre-training help avoid the vanishing gradient problem?

I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down ...
1
vote
1answer
256 views

Heterogenous layers in Neural Network

For this question when I say Neural Network I mean the standard Multi-layered Fully Connected Feed Forward Neural Network. I'm exploring the different activation functions and cost functions used in ...
2
votes
2answers
534 views

I cannot differentiate my loss function, what is the best method for optimizing the weights in my neural network?

Suppose the output of my network is $y \in [0, 1]$ for a given input x. The loss function to be minimized is $f(x) = -\Sigma (w_i y_i + | y_i - y_{i-1}|)$, with real weights $w_i$. The dependency ...
1
vote
1answer
310 views

FeedForward Alternative to Backpropagation

I am reading a blog post that tries to explain backpropagation. In the build up the author shows how a naive method for computing gradients is sub-optimal. Consider this: Naive feedforward ...
1
vote
0answers
293 views

Recursive backpropagation vs backpropagation

I recently read a paper that tried to find out what happened in a RNN by linearization around slow and fixed points. I can't figure out why we have to use linearization around these points. After I ...
0
votes
0answers
310 views

Why can't we backprop the gradients in the recurrent neural nets?

We use back propagation through time in the recurrent neural nets. It's done by adding up the gradients w.r.t to weights in each time step. But my question is why can't we just keep getting the ...
2
votes
1answer
2k views

Can someone please explain the truncated back propagation through time algorithm?

I am reading about RNNs and how to train them and I understood how back propagation works. I have the following model: $$ h_t=f(Ah_{t-1}+ B x_t),\\ \hat{y}_t=g(C h_t). $$ For a given sample $(x_1^T,...
1
vote
1answer
2k views

resilient backpropagation parameters selection

In the original paper for resilient backpropagation (http://paginas.fe.up.pt/~ee02162/dissertacao/RPROP%20paper.pdf), the author says "One of the main advantages of RPROP lies in the fact, that for ...
3
votes
1answer
1k views

How to interpret the classification boundary?

I am beginner to neural network and machine learning. I am working neural network with 1 hidden layer. I took spiral data set and I am trying to overfit the data. I applied neural network to it and I ...
4
votes
1answer
97 views

Probability theoretic analog to Neural Networks

I am trying to figure out how to do learning using probabilistic programming languages. For this I am following different paths to get a hold on the way of thinking. I understand modelling using ...
3
votes
0answers
94 views

Derive the gradients of a basic neural network

Given a neural network as following \begin{align*} &J = CE(y,\hat{y})=-\sum_i y_i log(\hat{y}_i)\\ &\hat{y} = softmax(z_2)\\ &z_2 = hW_2+b_2\\ &h = sigmoid(z_1)\\ &...
9
votes
1answer
3k views

Gradients for skipgram word2vec

I am going through the problems in the Stanford NLP deep learning class's written assignment problems http://cs224d.stanford.edu/assignment1/assignment1_soln I am trying to understand the answer for ...
4
votes
0answers
136 views

Backpropagation - computing partial derivative with respect to W

I am following a chapter on backprop derivation from the online book by Michael Nielsen In particular, following equation is derived in Chapter 2: ${∂C\over∂w^{l}_{jk}}=a^{l−1}_{k}δ^{l}_{j}$ Now, I ...
1
vote
0answers
33 views

does bp need to be simultanious? - XOR learning problem

I have a problem with XOR learning. I have 2(inputs),2,1(output) neuron neural network with sigmoidal function and normal sets ...
1
vote
1answer
126 views

Matrix-based implementation for an ANN with different number of nodes per hidden layer?

Given a neural network architecture that has multiple hidden layers and a different number of nodes in each hidden layer (assuming that this is a valid architecture), is it possible to implement feed ...
1
vote
0answers
70 views

Elaboration on weight change for output layer in neural network

I have trouble understanding the section regarding backpropagation in Murphy's Machine Learning book. He derives the weight change for the output layer (16.68) as follows: $\nabla_{\mathbf{w}_k} J_n ...
2
votes
2answers
292 views

Backpropagating a Dueling Architecture Network: Gradient Calculation

I started coding a Dueling Network Architectures for Deep Reinforcement Learning. I devided my network into two streams, arriving at a V(s) value and A(s,a) values. I arrived at the Q(s,a) output ...
-1
votes
3answers
566 views

Difference between cross-validation and back propagation

sorry if this is a dumb question. I want to know what the difference is? Mainly what how they're objectives differ from each other? I know that in cross validation you divide the training set into ...
4
votes
1answer
254 views

What is this math symbol used in a backpropagation tutorial: $\circ$

I was wondering what this unfilled circle meant. It is in this tutorial which implements a neural network from scratch. I have posted the specific part for your convenience. It is the line with $\...
0
votes
1answer
108 views

Backpropagation not incremental

Why is backpropagation not incremental? The definition of incremental would be to be able to update the weights with every single new data point. However, in stochastic gradient descent with a ...
3
votes
1answer
379 views

Backprop: Calc gradients for embed layers and how to do gradient check

First I want to check is I understand correctly how backprop works for NN with embed layers. Lets create simplest NN with embed layer Input Layer [i1; i2; i3; i4] - input nodes Embed layer [h1] [h2] ...
1
vote
1answer
510 views

What is the update rule for hidden layer if softmax activation function is used?

I am trying to understand how backpropagation works. I understood the basic concepts and became familiar with derivation of equations for sigmoid activation function. Specifically for hidden layers, ...
2
votes
1answer
243 views

CNN pooling and convolution

In the past, I've worked with the MNIST dataset and I am currently working with Java. I do not use any external libraries like numpy (for python) or something like that. I started from scratch and ...