Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
1
vote
1answer
21 views

Changing representation in deep neural network

Say I have a neural net that outputs a vector of length 4 such as: [0, 1, 2, 3] Now say that the only way to calculate the loss is to convert this output to a one-...
0
votes
0answers
11 views

Artificial neural networks as directed acyclic graphs (ANNs as DAGs)

My (perhaps somewhat superficial) impression is that various types of artificial neural networks (simple forward networks, convolutional networks, LSTMs, etc.) can be represented as directed acyclic ...
-1
votes
1answer
34 views

Neural network written from scratch learns much slower than PyTorch

I wrote a program from scratch using numpy for the feeding forward and backprogation of a network. I tested it against a program written using PyTorch with the same architecture and the network from ...
1
vote
0answers
34 views

What's wrong in this derivation of back-propagation errors?

I'm trying to find a rigorous derivation for the backpropagation algorithm, and I've gotten myself into something of a confusion. The confusion comes from when and why people transpose the weight ...
2
votes
0answers
60 views

Basic RNN sequence classifier diagram?

I'd like to build an RNN in numpy from scratch to really get come comfortable with backpropagation through time (BPTT.) In the below diagram and LaTeX, I show two neurons, each with a non-linearity, N(...
0
votes
0answers
13 views

Backpropagation through time for stacked RNNs

I was able to find the partial derivative of the cost function with respects to a single variable without much difficulty. However, this requires propagating backwards through the network for each ...
1
vote
0answers
14 views

Why do we fit Recurrent Neural Networks with backprop instead of message passing/expectation propagation?--as with hidden markov models

The form of a Recurrent Neural Network (RNN) seems to resemble that of a hidden markov model. With a hidden markov model we have transitions between discrete states, as well as an emission variable ...
0
votes
0answers
12 views

Why I think zero-centered activation function is no better than no zero-centered case? What's wrong with my understanding?

I read the answer in Why are non zero-centered activation functions a problem in backpropagation? I can understand that for a positive activation function, gradient of each dimension is of the same ...
0
votes
0answers
30 views

Initializing network weights to zero

Since my last question on the topic I have tried searching on my own how zero weight initialization impedes learning but I can't quite seem to wrap my head around the concept. The CS231n course notes ...
2
votes
1answer
38 views

Numerical computation of cross entropy in practice

The equation for cross-entropy is: $H(p,q)=-\sum_x{p(x)\log{q(x)}}$ When working with a binary classification problem, the ground truth is often provided to us as binary (i.e. 1's and 0's). If I ...
0
votes
0answers
10 views

Gradient check fails for some CNN filters

I've implemented a very basic single layer CNN from scratch in matlab which has a convolutional layer - RELU - linear classifier with soft max and cross entropy loss I used the gradient checking ...
3
votes
0answers
68 views

Levenberg-Marquardt algorithm NN training (backpropagation)

I'm currently looking into the different training methods of NN. I've implemented a simple gradient descent method using backpropagation in my NN (just a very simple NN with 1 hidden layer). I'm now ...
2
votes
1answer
49 views

clarification on back-propagation calculations for a fully connected neural network

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated. Let's say my fully connected neural network looks like ...
0
votes
1answer
36 views

Training of a deep Artificial Neural Network

I have few doubts related to training a neural network with more parameters (weights and biases) than number of data points. I know there exists discussion (on this platform) related to training such ...
2
votes
1answer
38 views

What type of functions can/cannot be handled by backprop?

I have a very basic question about backprop, which is what type of function it can and cannot calculate the gradient of, and whether if anyone have examples of such functions. I interpret backprop as ...
1
vote
0answers
47 views

Does the 'skip' in a residual network actually happen in the backward pass?

The following is a question regarding a residual block. Let $A^{[l-1]\{t\}}$ denote the activation of the $(l-1)^\mathrm{th}$ layer of a particular fully connected neural network, given the $t^\mathrm{...
0
votes
1answer
20 views

automatic diffentiation (autograd): when the explicit definition of the gradient function is needed?

In Pytorch and similar machine learning software, the Autograd module computes the gradient of a function without needing to explicit declare the derivative of each single function which composes the ...
0
votes
0answers
5 views

Does the number of hidden neurons of a neural network directly relate to the number of iterations until convergence?

Does the number of iterations until convergence relates to the the number of hidden neurons when I train a fully connected network with a single hidden layer and sigmoid activation functions using ...
3
votes
0answers
21 views

Does gradient descent assume updates of one layer/parameter at a time?

I read the following in "Deep Learning", from Goodfellow et al (Chapter 8, page 313): The gradient tells how to update each parameter, under the assumption that the other layers do not change. In ...
0
votes
0answers
24 views

Is my step by step derivation of quadratic cost function's (Neural Networks) partial derivative with respect to some weights matrix correct?

I am trying to revise the details of a Multi-layer Perceptron with a set of weight matrices $\mathcal W$ and a set of bias vectors $\mathbf b$. Here is the quadratic cost function I am using, $$C(\...
1
vote
1answer
21 views

ANN Cost Function Notation

I have been following This book on the fundamentals of NNs. It is currently outlining the MSE Cost function, and the Notation is tripping me up some. $$ C(w, b) = \dfrac{1}{2n} \sum_x \vert\vert y(x)...
1
vote
1answer
45 views

Actual impact of W=0 in a neural network

As far as I understand, one of the main claimed problems with initializing e.g. a feed-forward neural network (with several $\text{tanh}$ or $\text{ReLU}$ layers) with $W=0$ is that it doesn't break "...
1
vote
0answers
25 views

Is it possible and if so how to build a neural network such that it doesn't backpropagate in certain regions of the NN?

I had an idea to improve a neural network I'm currently using, but I'm quite new to machine learning so I don't know if it's possible to implement or how difficult it is or simply if isn't worth. The ...
3
votes
0answers
18 views

Benefits of saturation in activation functions

It's known that saturation of activation functions in neural networks leads to vanishing gradients or dead units, so modern practice often avoids them, instead opting for e.g. ReLUs, Leaky ReLUs and ...
0
votes
0answers
13 views

How good is batch normalization in avoiding vanishing/exploding gradients?

Incase of deep neural network with many layers, how good is batch normalization in avoiding vanishing/exploding gradients. Consider the problem is only due to too many layers(too many multiplications ...
0
votes
2answers
52 views

Maths for deeply understanding backpropgation

I have been trying to develop a deeper understanding of Neural Networks so I can understand the libraries such as tensorflow and others. I have had good success with pereceptron models, and have a ...
0
votes
0answers
34 views

Neural Network: Why is Training and Testing giving different results for same input only for XOR, when it works for AND and OR?

I tried to implement a ANN in Java which does not have the Network acting as a master managing values, but the vertices and edges evaluate their values and pass them forward when the network tries to ...
0
votes
0answers
29 views

How to represent the bias node in simple MLP?

I have a simple 2-2-1 fully connected network, which I suppose means it does not include bias nodes since you don't connect bias nodes at every layer. I have two sets of weights for each layer ...
0
votes
0answers
3 views

Input gradient in convolutional backpropagation

(Note: this is distinct from my previous question) My 2D convolution is defined as follows: $$y_{i,j} = \sum_{m=0}^{f_w-1}\sum_{n=0}^{f_h-1}x_{si+m,sj+n}f_{m, n}$$ where $s$ is the stride (this ...
1
vote
0answers
14 views

Gradient in Convolutional Layer

I have a convolutional neural network that operates on $4$-tensors. I'm trying to calculate the gradient w.r.t. the 4-dimensional filter, i.e. $\frac{\partial E}{\partial f}$, given the gradient w.r.t....
1
vote
1answer
50 views

If L2 regularization parameter is high and learning rate low, can cost of cross entropy loss function increase?

I coded a neural network from scratch. When regularization parameter is too high and the learning rate too low, cost increases. I suspect that the added cost (associated with regularization) to loss ...
3
votes
1answer
25 views

Backpropagation demo gives always the same result after training

Just for practice, i've tried to create a super-basic neural network script (one neuron for each layer, didn't want to deal with matrices for the sake of simplicity, not that I don't know how to). I'...
0
votes
0answers
23 views

The Graph Neural Network- understanding the back propagation mechanism

I am having trouble understanding the derivation of the back propagation algorithm for graph neural networks, as derived in Scarselli 2009 "The Graph Neural Network Model." (IEEE Transactions on ...
0
votes
0answers
23 views

Softmax backpropagation

I know there are similar questions out there, but none of the answers really helped me. I'm working on an own neural network implementation and I want to implement the softmax activation function. I'm ...
1
vote
0answers
29 views

BackPropagation and Flatten layer in CNN

everybody. I'm trying to create CNN(Convolutional Neural Network) without frameworks(such as PyTorch,TensorFlow,Keras and so on) on Python. Who don't know or forgot what is exactly CNN is: To ...
0
votes
0answers
12 views

Simple ANN model converges with tanh(x) as the activation function, but it doesn't with leaky ReLu

I'm training a simple ANN model (MLP) using as the activation function tanh(x) and, after some interactions, it converges with error equal to 10^-5, here's my full ...
0
votes
0answers
15 views

Weights between the Last 2 layers keep getting negative

TL;DR weights between the last 2 layers keep getting negative to the point that the softmax(z) of the output layer can't divide by zero ( e^-750 ~= 0 thus deciding by 0) I am making a Neural Network ...
1
vote
0answers
41 views

Weight Clipping range with WGAN and relation with other factors

For a while, my code with WGAN has failed to generate quality images through a complicated multi-class database. My issue has been with implementation and not code. Recently I read the WGAN-GP paper ...
0
votes
0answers
19 views

In backprop/gradient descent, why isn't the reciprocal of gradient used when updating weights?

I've seeing almost all tutorials on backprop stating the following for weight update: $$ W_n = W_{n-1} - lr \times L\frac{dL}{dW} $$ Since $\frac{dL}{dW}$ is the influence of $W$ on $L$, and we are ...
0
votes
0answers
13 views

Optimizing neural networks for maximum utilization of data instead of faster training

A number of methods, some of which I'm familiar with, some not, are used to optimize neural networks for faster convergence. Tricks like batching, different backpropagation algorithms, and so on speed ...
0
votes
0answers
17 views

Neural network with back propagation change of weights with initialization equal zero

I'm creating a neural network for school project and I have a question. I have created a 4x3x1 neural network with a bias both in the first and in the second layer (equal to 1) and with tanh as ...
2
votes
1answer
16 views

What is the dimensionality of the cost function with this specific ANN structure?

I have the following ANN architecture, the neuron is a sigmoid neuron: Where the weight and parameter matricies are given by: $$ \begin{vmatrix} & x1& & x2& & x3& \end{...
3
votes
2answers
652 views

Neural nets: How to get the gradient of the cost function from the gradient evaluated for each observation?

The gradient of the cost function, $E$, changes for each input observation. I have taken $E$ to be the sum of least squares error, for example. To see this, note that the partial derivative with ...
3
votes
0answers
88 views

Why not to initialize a neural network's weights to zero? [duplicate]

The notes for Stanford's online course on CNN's mention not to initialize all the weights to zero, because: … if every neuron in the network computes the same output, then they will also all ...
0
votes
1answer
29 views

Back-propagation Algorithm

Why does Backpropagation Algorithm backpropagate a value back on a neuron with activation zero which can't have an influence on the error ? I assume binary activations of the neurons. ...
0
votes
0answers
4 views

Relation of TBPTT and Saving States

I was wondering if someone could check whether my understanding of Truncated Backpropagation Through Time (TBPTT) is correct, maybe even with particular focus on Tensorflow. Let us assume I have very ...
0
votes
0answers
22 views

How is the Cost Function of a neural network defined?

I was reading Michael Nielson's book on Neural Networks in which he writes that: The first assumption we need is that the cost function can be written as an average $C=\sum_x{C_x}$ over cost ...
0
votes
0answers
15 views

How to derive mathematically that derivative of |Ax-y|^2 with respect to A is 2|Ax-y| x^T [duplicate]

How to get transpose part when derive mathematically $$ \frac {\partial|Ax-y|^2}{\partial A} = 2|Ax-y|x^T $$
1
vote
1answer
31 views

Backpropagation in neural network

I have been trying to better understand backpropagation, so I decided to try to derive it for myself, but there is one step I'm not totally sure about. First explaining my notation $C_0$ is the loss ...
1
vote
0answers
34 views

Question about Connectionist Temporal Classification (CTC) gradient

I have read the original CTC paper by Graves et al, but am confused about equation 16, for which the authors derive the gradient of the negative log likelihood objective with respect to the inputs of ...

1
2 3 4 5
8