Questions tagged [backpropagation]

Backpropagation, an abbreviation for "backward propagation of errors", is a common method of training artificial neural networks used in conjunction with an optimization method such as gradient descent.

Filter by
Sorted by
Tagged with
5
votes
1answer
1k views

Transfer learning: How and why retrain only final layers of a network?

In this video, Prof. Andrew Ng says regarding transfer learning: Depending on how much data you have, you might just retrain the new layers of the network, or maybe you could retrain even more ...
5
votes
2answers
2k views

What sort of problems is backpropagation best suited to solving, and what are the best alternatives to backprop for solving those problems?

I am developing a neuroscience-inspired training algorithm for feed-forward neural networks. The natural benchmark for comparison is backpropagation. So I need to know to know what sort of ...
5
votes
1answer
860 views

What is the difference among stochastic, batch and mini-batch learning styles?

So far as I know, we have the following scenario: stochastic: The error is calculated for each sample s. So, we can calculate the gradients for s. And we can update the weights of the network ...
5
votes
1answer
100 views

Probability theoretic analog to Neural Networks

I am trying to figure out how to do learning using probabilistic programming languages. For this I am following different paths to get a hold on the way of thinking. I understand modelling using ...
5
votes
1answer
2k views

Is “batch normalization” applied for output layer as well?

batch normalization in a sense that in a given layer, you standardize the neurons' values, then multiply each with some trainable scaling constant, and shift them with some another trainable shifting ...
5
votes
1answer
1k views

Back-propagation in Convolution layer

Most examples I found on the internet explain well back-propagation in convolution layer, but only with a single kernel and single input channel. I do not understand how to do back-propagation for ...
5
votes
1answer
4k views

Truncated Back-propagation through time for RNNs

I am not very clear on what is the proper way to train an RNN. Suppose we are using a vanilla RNN and are given some categorical sequence $x$ of length $T$: $$x= [ x_1,\ldots,x_T]$$ To fit the ...
5
votes
1answer
3k views

Sigmoid activation hurts training a NN on pyTorch

I'm a beginner in the field of Machine Learning and I'm currently trying to get my hands "dirty" for the first time with some code after completing a course in that field. I'm using pyTorch to train ...
5
votes
1answer
556 views

How does co-adaptation occur in deep neural nets

From what I can understand, it describes the phenomenon of when neurons detect the same features. Why does this happen?
5
votes
0answers
327 views

Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:...
4
votes
1answer
262 views

What is this math symbol used in a backpropagation tutorial: $\circ$

I was wondering what this unfilled circle meant. It is in this tutorial which implements a neural network from scratch. I have posted the specific part for your convenience. It is the line with $\...
4
votes
1answer
983 views

Why does SGD and back propagation work with ReLUs?

ReLUs are not differentiable at the origin. However, they are widely used in Deep Learning together with Stochastic Gradient Descent algorithms and Backpropagation, where the gradients of the loss ...
4
votes
1answer
5k views

How to update weights in a neural network using gradient descent with mini-batches?

How does gradient descent work for training a neural network if I choose mini-batch (i.e., sample a subset of the training set)? I have thought of three different possibilities: Epoch starts. We ...
4
votes
2answers
4k views

backpropagation - bias nodes and error

I am implementing the stochastic gradient descent version of backpropagation from Tom Mitchell's Machine Learning book which has the steps for each training instance $\langle\vec{x},\vec{t}\rangle$: ...
4
votes
1answer
2k views

Question with Matrix Derivative: Why do I have to transpose?

In the equation for Recurrent Neural Networks: $$ h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b) $$ Where $h_t$ is of size (N,H) Where $W_{hh}$ is of size (H,H) Where $W_{xh}$ is of size (D,H) Where $...
4
votes
1answer
53 views

What are the practical uses of Neural ODEs?

"Neural Ordinary Differential Equations", by Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud, was awarded the best-paper award in NeurIPS in 2018 There, authors propose the ...
4
votes
1answer
373 views

Transition from “old-school” neural network methods to deep learning?

As far I know the current state of deep learning favours a rather simplistic setup -- in short: many layers to allow for representational learning, maxout or a similarly suited activation function to ...
4
votes
1answer
276 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
4
votes
3answers
4k views

In Neural Network back propagation, how are the weights for one training examples related to the weights for next training examples?

In Simple Neural Network back propagation, we normally use one round of forward and back propagation in every iteration. Let's assume, we have one training example for any arbitrary dimensions, and ...
4
votes
3answers
875 views

Back propagation in Convolutional neural networks

I am trying to understand how CNN works. I want to use them in object recognition task. I thouhgt that CNN is unsupervised networks. My main question is how can I implement the back propagation ...
4
votes
2answers
1k views

Why do Deep Learning libraries force the cost function to output a scalar?

Let's say we have a neural net with: 5 input neurons some arbitrary amount of hidden layers 3 output neurons Let's say we're using minibatches of size 32. So, if we input a 5x32 matrix into the ...
4
votes
1answer
674 views

Can all neural network with DAG topology be trained by Back-prop?

Can all neural network having directed acyclic graph (DAG) topology be trained by back propagation methods? You can assume that the activation functions of all neurons are differentiable. I mean by ...
4
votes
1answer
2k views

Choosing time points to run backpropagation through time

Suppose we're training a recurrent neural net (RNN) on a single, long time series using truncated backpropagation through time (BPTT). We make repeated sweeps through the time series, updating ...
4
votes
1answer
932 views

What is the most efficient method to handle long time sequences (LSTM)?

I am using LSTM and I have several long time sequences of varying length. Most of them are about 6,000-7,000 timesteps on average, but several are around 40,000 long. I am not sure which of this would ...
4
votes
1answer
3k views

Batch normalization: How to update gamma and beta during backpropagation training step?

The backpropagation step of batch normalization computes the derivative of gamma (let's call it dgamma) and the derivative of <...
4
votes
0answers
574 views

ReLU derivative - second order effects

I am reading the Deep Learning Book, where there is a section on generalisations of the ReLU (section 6.3.1). It states: The second derivative of the rectifying operation is 0 almost everywhere, ...
4
votes
0answers
136 views

Backpropagation - computing partial derivative with respect to W

I am following a chapter on backprop derivation from the online book by Michael Nielsen In particular, following equation is derived in Chapter 2: ${∂C\over∂w^{l}_{jk}}=a^{l−1}_{k}δ^{l}_{j}$ Now, I ...
4
votes
2answers
1k views

Backpropagation proof and usage confusion

I've been taking Andrew Ng's course on Coursera, and although it has been great so far, I loathe his lack of supplementary documents on proofs. Thankfully, there are some great articles found pretty ...
3
votes
4answers
6k views

How to speed up training of a Neural Network?

I'm writing a thesis where I developed a script that generates NN and precalculates weights and biases to reduce a required number of epochs when I train a network. In my work, using examples I ...
3
votes
2answers
5k views

MNIST digit recognition: what is the best we can get with a fully connected NN only? (no CNN)

To fully understand how it works internally, I'm re-writing a neural network from scratch in Python + numpy only. (As it's for learning purposes, performance is not an issue). Before moving to ...
3
votes
2answers
2k views

Does a Neural Network actually need an activation function or is that just for Back Propagation?

I have a feed forward neural network (1 hidden layer with 10 neurons, 1 output layer with 1 neuron) with no activation function (only transfer by weight + bias) that can learn a really wonky sin wave (...
3
votes
1answer
263 views

Why is this the correct formula to update the NN weights in Q-learning?

I'm trying to implement Q-learning to train an AI bot to play Pokemon battles. Since there is a large state space (corresponding to all possible states a battle can have in between moves), I can't use ...
3
votes
1answer
1k views

How to interpret the classification boundary?

I am beginner to neural network and machine learning. I am working neural network with 1 hidden layer. I took spiral data set and I am trying to overfit the data. I applied neural network to it and I ...
3
votes
4answers
2k views

Do extra hidden layers prevent convergence?

I have designed a simple feed-forward neural network using stochastic gradient descent. I use 22 inputs, 4 hidden layers, 1 output and am using a learning rate of 0.7 and momentum of 0.3. I have about ...
3
votes
1answer
142 views

Mean or sum of gradients for weight updates in SGD

I am using single observation to compute losses using neural network implementation in PyTorch. I am confused in a small detail of SGD. If I compute loss and do ...
3
votes
1answer
6k views

Mean Absolute Error (MAE) derivative

$MAE=|y_{pred} - y_{true}|$ $\dfrac{dMAE}{dy_{pred}} = ?$ I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some ...
3
votes
1answer
3k views

What is the significance of the Delta matrix in Neural Network Backpropagation?

I'm currently taking Andrew Ng's Machine Learning course on Coursera, and I feel as though I'm missing some key insight into Backpropagation. Particularly, I'm stuck on this algorithm slide: First, ...
3
votes
2answers
2k views

Matrix and Vector Approaches to Backpropagation in a Neural Network

I recently implemented a neural network, with backpropagation in a fully matrix approach, as described here, where the whole dataset is used for each backprop: http://ufldl.stanford.edu/wiki/index.php/...
3
votes
2answers
565 views

Why does pre-training help avoid the vanishing gradient problem?

I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down ...
3
votes
2answers
3k views

Numerical check of gradient in neural network

I am trying to check if my implementation of backpropogation is correct by checking the calculated gradients with the numeric gradient. I am testing it on a very simple linear network (i.e. no ...
3
votes
1answer
148 views

How exactly is the error backpropagated in backpropagation?

I am reading a book on neural networks, and am now doing a chapter on backpropagation. (See chapter here). In this chapter, the writer is presenting four equations, that together form the backbone of ...
3
votes
1answer
167 views

Higher Order of Vectorization in Backpropagation in Neural Network

I am learning a machine learning class online from Stanford, namely CS 229. There is one section about deep learning and back-propagation in deep learning. The network looks like: The forward ...
3
votes
1answer
190 views

What is the reason that reduce training time over epoch for LSTM?

I am training and recurrent neural network and observed less time is needed over time. What could be the reason? I would think calculating the gradient, and update the parameters in the network would ...
3
votes
1answer
6k views

Backpropagation with Cross-entropy Cost Function

I'm using the cross-entropy cost function for backpropagation in a neutral network as it is discussed in neuralnetworksanddeeplearning.com. I got help on the cost function here: Cross-entropy cost ...
3
votes
2answers
1k views

Full batch backpropagation implementation

I am trying to wrap my head around using batch backprop in a neural network. I have a very code-oriented mind, and I'm trying to figure out whether it's possible to parallelize the full batch ...
3
votes
1answer
407 views

Is backpropagation algorithm same for both full-connected and local-connected neural network?

Is the backpropagation (BP) algorithm the same for both fully-connected and locally-connected (or partially-connected) neural networks? I know how to use BP for a fully-connected network, but I don't ...
3
votes
1answer
184 views

Backpropagation in capsule networks

Trying to create a capsule network implementation, I've browsed through several tutorials and code sources, but was unable to find how back-propagation for capsule networks is implemented. It is not ...
3
votes
1answer
268 views

Understanding a proof of conditions for vanishing/exploding gradient in RNNs

I'm looking at some of the preliminaries in understanding vanishing/exploding gradients with recurrent neural networks (RNNs), and I see this paper referenced quite a lot: https://arxiv.org/abs/1211....
3
votes
1answer
845 views

Why zero-centered output affected the backpropagation?

I read the answer in Why are non zero-centered activation functions a problem in backpropagation? but I still can't understand. Assume$$f=\sum w_ix_i+b$$ $$\sigma(x)=\dfrac{1}{1+e^{-x}}$$, and loss ...
3
votes
2answers
1k views

How to update kernel values in a convolutional layer during backward pass?

I started coding backpropagation for a simple convnet and had some troubles understanding the algorithm. I do get the idea of weight update based on gradients, but because the filter kernel parameters ...