Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

-1
votes
0answers
9 views

Backpropagation for ReLU and binary cross entropy

I'm trying to get the value of the gradient on the penultimate layer where the output is activation function ReLU. I have a logistic function on the last layer. I got the gradient value on the last ...
0
votes
0answers
23 views

Understanding backprop equations [on hold]

I was watching a video on backprop from deeplearning.ai where one particular thing confused me a lot. In the backprop, as shown below, Why aren't we averaging <...
0
votes
0answers
13 views

XOR Neural Network, Problem finding shapes of delta for backpropagation algorithm

I am taking the Machine Learning course by Andrew Ng on coursera. I am trying to make a neural network learn to do XOR, but I am facing a problem regarding the shapes of the $\delta$ vectors, and $\...
2
votes
1answer
18 views

Calculating the Policy Gradient for a Monte Carlo REINFORCE Algorithm

I am currently trying to implement the Monte Carlo REINFORCE algorithm, as described in Sutton and Barto's book Reinforcement Learning (p. 328, Second Edition). If $\theta$ denotes the parameter for ...
0
votes
0answers
21 views

Vector dimension mismatch with back propagation [on hold]

I'm making a neural network from scratch, and struggling with back prop. I understand how it works in principle, but when it comes to code, everything seems to fall apart. Here's my architecture: ...
5
votes
2answers
67 views

Can $\sin(x)$ be used as activation in deep learning?

$\sin(x)$ seems to zero centered which is a desirable property for activation functions. Even the gradient won't vanish at any point. I am not sure if the oscillating nature of the function or its ...
0
votes
0answers
19 views

How to handle maxpool layer backpropagation with recurring max values in same position

Say I have a layer a: 3 4 2 1 5 0 8 6 4 The maxpool using 2x2 filter is: <...
0
votes
1answer
24 views

Meaning of “backpropagate through Gaussian distributions”?

I just started reading about GAN theory properly for the first time and I have a question about a comment in the original GAN paper. On page two there's a paragraph that states the following: ... ...
1
vote
1answer
11 views

How does backpropagation work with mixed architectures?

This question raised to me since I'm unsucessfully training a CNN-LSTM network atm. If for instance, LSTM requires a different type of BP algorithm (TBPTT), how do softwares deal with it? What is the ...
0
votes
0answers
10 views

Clarification on notation used to present back propagation algorithm in 'The Deep Learning Book'

In the deep learning book (free version is available online) the backpropation algorithm is explained in section 6.5. I have a question on equation (6.53): $$\frac{\partial u^{(n)}}{\partial u^{(j)}}...
0
votes
1answer
47 views

How is the loss(Backpropagation) for simple RNN calculated when dealing with batch?

I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself. However, it seems I have not ...
0
votes
0answers
16 views

Backpropagation - Assumption or Error in Bishop's “PRaML”

I am currently reading Bishop - "Pattern Recognition and Machine Learning" (2006) and I could not figure out if I missed an assumption he made or if it is just wrong. In the chapter about ...
0
votes
0answers
21 views

backpropagation for multiple hidden layers

I'm currently implementing a small neural network library to learn the concepts coming with them better. As probably most people do, I'm struggling a bit with the backpropagation algorithm, especially ...
0
votes
0answers
28 views

Training convolutional neural network

I implemented Convolutional Neural Network from scratch for image recognition for 5 classes. When I train it using only one image from one class it seems to be working, because accuracy for this class ...
0
votes
0answers
27 views

Calculation of backpropagation of a NN with skip nodes

Assume this neural net. In our course we had to provide the forward and backward pass for this net. This is the solution we received: Even with the solutions provided I still have problems ...
0
votes
0answers
18 views

What are the advantage of using Sigmoid and Softmax and disadvantageous of both? [duplicate]

I am trying to understand the architecture of the neural network. I am supposed to make decision of how many hidden layers and what activation functions to use in the hidden layer and which activation ...
0
votes
0answers
10 views

ANN for Boundary Value Problem

I have a question regarding solving Boundary Value Problems (BVP) using ANNs. My understanding is that this is currently a challenging task. Most scientific literature on the subject is interested in ...
0
votes
2answers
121 views

Softmax layer derivative by hand

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=...
0
votes
0answers
35 views

Multiple filters during backpropagation in convolutional neural network

Let's say we have an Input 10x10x3 (WxHxD) and 5 filters 3x3x3. Convolution between Input and filters will be 8x8x5. During backpropagation we will get error with the same size 8x8x5. While ...
2
votes
2answers
60 views

How does training affect the norm of weight matrices?

I have a neural network $F(W,x): \mathbb{R}^d \rightarrow \mathbb{R}^k$ with $L$ layers, $m$ neurons per layer, ReLu activation, softmax on the last layer and $n$ datapoint. My loss function is the ...
1
vote
1answer
64 views

Derivative of the loss function w.r.t to X for the backpropagation

I would like to ask you why do we need to calculate a derivative of the loss function w.r.t X? It seems like, that for the backpropagation we need to calculate only a derivative w.r.t W. Can you ...
0
votes
0answers
26 views

Backpropagation through LSTM and MLP

For didactic reason, I am currently implementing in numpy an LSTM network for classifications. I need to add on top of the LSTM another fully connected layer, because I don't want the output to have ...
0
votes
1answer
31 views

Neural Networks - Back Propagation and Perceptrons

While studying about neural networks (still on basics - not Deep Learning etc.,) two questions came on my mind. What is the reason for replacing the hard limiter function in the nodes of the ...
0
votes
0answers
20 views

Mini Batch Gradient Descent Backpropapagation

I am a beginner to machine learning. I have derived the equations for backpropagation, and for the weight update for hidden layers, the update rule uses the output vector of the layer to multiply with ...
1
vote
0answers
21 views

Why is Backpropogation used instead of Rosenblatt's learning Algo or gradient descent to train MLP's?

In roesnblatt's learning algo and gradient descent the output is calculated for each input and based on the error b/w the outputs calculated and desired outputs the weights are updated. Why is ...
0
votes
0answers
21 views

Performing backpropagation on a 2 layer neural network

I am attempting to construct an NN from scratch without vectorizing. This is the network I'm attemping to model, where h1 and h2 represent the hidden layer nodes (h1 being the top, h2 being the ...
1
vote
0answers
53 views

Can you train deep recurrent neural network layer by layer?

Specifically for Gated Recurrent Unit, and say GRU is "layered" via but suppose it's only 2 layers deep for simplicity, and suppose the "total loss" = $L$ = $\sum{l_{t}} = \sum{error(y^{2}_{t})}$ ...
0
votes
1answer
23 views

How are RNNs with inputs greater than the defined sequence length implamented

To clarify the slightly ambiguous language in the title. I have an RNN (actually 2 stacked RNN layers) that take input X of size X [batch_size, sequence_length, features] the model is trying to ...
0
votes
0answers
24 views

Question about PyTorch tutorial

In this PyTorch tutorial the backprop to compute gradients is shown with the following code: ...
4
votes
3answers
105 views

how to make sense of the number of observations per parameters in deep learning models?

In a simple linear regression setting, it is common to talk about a minimum number of observations per parameter (which characterise the the degree of freedom). And it is easy to see that for multiple ...
2
votes
1answer
1k views

Cross Entropy Loss for One Hot Encoding

CE-loss sums up the loss over all output nodes $\sum_i[ - target_i*\log(output_i) ]$. The derivative of CE-loss is: $- \frac{target_i}{output_i}$. Since for a target=0 the loss and derivative of ...
0
votes
0answers
30 views

Backpropagation formula for simple ANN

I'm trying to solve an exercise. I'm asked to train a 2-layer neural network, given a small dataset. Below is a picture I drew to understand the architecture and derive the formula for backpropagation....
1
vote
1answer
18 views

Storage and re-computation of Intermediate / Weight / Back-propagated Gradients

I need to track the computation and storage of different parts of my network training. To be on the same page, let's assume the simple following scenario (biases omitted) Questions Local Gradients - ...
2
votes
1answer
36 views

Backpropagation wrong? Doesn't it update dependent variables in hidden layer

In a multi layer perceptron or feedforward neural network, isn't backpropagation updating weights of the middle layers that are dependent variables? So for a particular hidden layer, it calculates all ...
3
votes
2answers
976 views

MNIST digit recognition: what is the best we can get with a fully connected NN only? (no CNN)

To fully understand how it works internally, I'm re-writing a neural network from scratch in Python + numpy only. (As it's for learning purposes, performance is not an issue). Before moving to ...
1
vote
2answers
158 views

Why is the second derivative required for newton's method for back-propagation?

I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely. I have seen this same question, and the widely accepted answer ...
1
vote
1answer
62 views

Using step function as activation function in the final layer

I am using variational autoencoders as machine learning algorithm. My input data are images/matrices that represent user interface layouts or how the HTML page will be divided. I am thinking to ...
0
votes
0answers
15 views

Can adaptive learning rate method be used for dropout regularization?

if the neurons are deactivated randomly for each forward pass during an iteration, Can adaptive learning rate method for neural network such as RMSprop be used for the case of dropout regularization?
0
votes
1answer
308 views

Does dropout regularization prevent overfitting due to too many iterations?

For image classification problem, let's say, and given a neural network to train on, if you were to run too many iterations for a single image of a cat would not generalize well into other images of ...
0
votes
0answers
30 views

Backpropagation NN - is it trained?

I am working on a simple Back propagation Neural Network that is predicting next value of a sensor reading. It took me a while to train it based on the last year data and now i have the results. ...
2
votes
1answer
63 views

Backpropagation in a logarithmic layer of a regression NN

A "logarithmic neuron" is defined as follows [1]: Which for inputs $\left\{ {{x_1},...,{x_n}} \right\}$ yields an output of $z=\prod\limits_{i = 1..n} {x_i^{{w_i}}}$ (in MATLAB, the activation ...
1
vote
0answers
21 views

Batch learning in digits recognition (MNIST database) [duplicate]

While working my way through M. Nielsen's "Neural networks and deep learning", I decided to try out some presumably silly things to really understand why they won't work and/or why it's not a good ...
3
votes
1answer
336 views

Is “batch normalization” applied for output layer as well?

batch normalization in a sense that in a given layer, you standardize the neurons' values, then multiply each with some trainable scaling constant, and shift them with some another trainable shifting ...
0
votes
1answer
57 views

back propagation in neurons with zero weight and some specific conditions

I have read a lot of articles to understand what is happening behind the scene in backpropagation like Ive gone through this and many other like that. I think I understand how the backpropagation ...
0
votes
1answer
121 views

How are biases updated when 'batch size' > 1?

This is my network represented in matrices: (a dot represents an arbitrary number) Feed-forwarding: (I omitted nesting it all in an activation function for the sake of brevity) Backpropagation The ...
1
vote
1answer
45 views

Validate implementation of back-propagation algorithm

Let's say I implemented a CNN. Is there an easy way I can validate, that my implementation of back-propagation does not contain errors ? May be I can feed some dummy values into my network so it can ...
1
vote
1answer
31 views

What does “learn the linear part of a mapping” mean?

In the paper "Efficient BackProp" , the authors talk about initializing the weights not too small and not too large: Intermediate weights that range over the sigmoid's linear region have the ...
2
votes
1answer
61 views

Higher Order of Vectorization in Backpropagation in Neural Network

I am learning a machine learning class online from Stanford, namely CS 229. There is one section about deep learning and back-propagation in deep learning. The network looks like: The forward ...
1
vote
1answer
203 views

Why can't we use backpropagation and gradient descent on a Restricted Boltzmann Machine

Can someone please explain why we cannot use the backpropagation algorithm and gradient descent to train a Restricted Boltzmann Machine. In other words, why can't we train an RBM in the same manner ...
4
votes
1answer
60 views

What is the reason that reduce training time over epoch for LSTM?

I am training and recurrent neural network and observed less time is needed over time. What could be the reason? I would think calculating the gradient, and update the parameters in the network would ...