All Questions

Filter by
Sorted by
Tagged with
0
votes
0answers
36 views

Backprop through un-differentiable function f(x)?

Let's say I have a vector of gradients (ie. dL / dy). These gradients are the result of taking the derivative of the Loss function with respect to the output of a ...
0
votes
0answers
14 views

CNN Backpropagation Clarrification

Hi I am just trying to make sure my understanding of backpropagation with CNNs is correct, specifically CNNs that have multiple filters in each layer. This is how I have implemented backpropagation ...
0
votes
0answers
10 views

Why cache gradients for params between training examples?

I was going through Karpathy's guide here where he defines an simple multiplication gate's forward and backward passes like so ...
0
votes
0answers
16 views

How to compute weight change for hidden layers with cross-entropy loss? [duplicate]

I'm trying to train a neural net with 1 hidden layer (RELU) softmax output layer cross-entropy loss stochastic gradient descent My implementation seems to work fine when I don't use any hidden ...
0
votes
0answers
25 views

What's the bug in my implementation/understanding of backpropagation?

For learning purposes, I'm trying to implement a simple neural network with only linear layers followed by logistic activation. As far as I understand, the backpropagation algorithm exploits the ...
2
votes
1answer
94 views

Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable ...
1
vote
1answer
19 views

Training Perceptrons with Backprop

Is it possible to train a simple perceptron with a threshold activation function such as this one: https://en.wikipedia.org/wiki/Perceptron with Backpropagation instead of the perceptron rule? is it ...
0
votes
1answer
35 views

Confusion when Learning Parameters in BAYESIAN MODELS

I'm learning Bayesian Models but i still have some issues with the training of the parameters. These are my two questions : 1) Recall the Bayesian formula : $$p(\theta|X) = \frac{ p(X|\theta) \; p(\...
0
votes
3answers
48 views

Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated ...
0
votes
1answer
22 views

Is backpropagation is used in validation data set? [closed]

Hello guys I am very confused as I am building a deep learning image classifier from raw python code ,so my question is that:-is backpropagation used in validation set to get the model more accuracy ...
0
votes
0answers
16 views

XOR Neural Network, Problem finding shapes of delta for backpropagation algorithm

I am taking the Machine Learning course by Andrew Ng on coursera. I am trying to make a neural network learn to do XOR, but I am facing a problem regarding the shapes of the $\delta$ vectors, and $\...
0
votes
0answers
31 views

How to handle maxpool layer backpropagation with recurring max values in same position

Say I have a layer a: 3 4 2 1 5 0 8 6 4 The maxpool using 2x2 filter is: <...
0
votes
1answer
31 views

Backpropagation - Assumption or Error in Bishop's “PRaML”

I am currently reading Bishop - "Pattern Recognition and Machine Learning" (2006) and I could not figure out if I missed an assumption he made or if it is just wrong. In the chapter about ...
0
votes
0answers
251 views

backpropagation for multiple hidden layers

I'm currently implementing a small neural network library to learn the concepts coming with them better. As probably most people do, I'm struggling a bit with the backpropagation algorithm, especially ...
0
votes
0answers
31 views

What are the advantage of using Sigmoid and Softmax and disadvantageous of both? [duplicate]

I am trying to understand the architecture of the neural network. I am supposed to make decision of how many hidden layers and what activation functions to use in the hidden layer and which activation ...
0
votes
0answers
36 views

Backpropagation through LSTM and MLP

For didactic reason, I am currently implementing in numpy an LSTM network for classifications. I need to add on top of the LSTM another fully connected layer, because I don't want the output to have ...
0
votes
0answers
36 views

Mini Batch Gradient Descent Backpropapagation

I am a beginner to machine learning. I have derived the equations for backpropagation, and for the weight update for hidden layers, the update rule uses the output vector of the layer to multiply with ...
1
vote
0answers
75 views

Can you train deep recurrent neural network layer by layer?

Specifically for Gated Recurrent Unit, and say GRU is "layered" via but suppose it's only 2 layers deep for simplicity, and suppose the "total loss" = $L$ = $\sum{l_{t}} = \sum{error(y^{2}_{t})}$ ...
0
votes
1answer
38 views

How are RNNs with inputs greater than the defined sequence length implamented

To clarify the slightly ambiguous language in the title. I have an RNN (actually 2 stacked RNN layers) that take input X of size X [batch_size, sequence_length, features] the model is trying to ...
3
votes
2answers
2k views

MNIST digit recognition: what is the best we can get with a fully connected NN only? (no CNN)

To fully understand how it works internally, I'm re-writing a neural network from scratch in Python + numpy only. (As it's for learning purposes, performance is not an issue). Before moving to ...
1
vote
2answers
269 views

Why is the second derivative required for newton's method for back-propagation?

I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely. I have seen this same question, and the widely accepted answer ...
0
votes
0answers
16 views

Can adaptive learning rate method be used for dropout regularization?

if the neurons are deactivated randomly for each forward pass during an iteration, Can adaptive learning rate method for neural network such as RMSprop be used for the case of dropout regularization?
1
vote
0answers
21 views

Batch learning in digits recognition (MNIST database) [duplicate]

While working my way through M. Nielsen's "Neural networks and deep learning", I decided to try out some presumably silly things to really understand why they won't work and/or why it's not a good ...
3
votes
1answer
1k views

Is “batch normalization” applied for output layer as well?

batch normalization in a sense that in a given layer, you standardize the neurons' values, then multiply each with some trainable scaling constant, and shift them with some another trainable shifting ...
1
vote
1answer
32 views

What does “learn the linear part of a mapping” mean?

In the paper "Efficient BackProp" , the authors talk about initializing the weights not too small and not too large: Intermediate weights that range over the sigmoid's linear region have the ...
3
votes
1answer
98 views

Higher Order of Vectorization in Backpropagation in Neural Network

I am learning a machine learning class online from Stanford, namely CS 229. There is one section about deep learning and back-propagation in deep learning. The network looks like: The forward ...
3
votes
1answer
96 views

What is the reason that reduce training time over epoch for LSTM?

I am training and recurrent neural network and observed less time is needed over time. What could be the reason? I would think calculating the gradient, and update the parameters in the network would ...
3
votes
1answer
207 views

Why is this the correct formula to update the NN weights in Q-learning?

I'm trying to implement Q-learning to train an AI bot to play Pokemon battles. Since there is a large state space (corresponding to all possible states a battle can have in between moves), I can't use ...
1
vote
0answers
350 views

Need help writing a neural network for a Pokemon battle

I'm trying to write a neural network that's able to select the optimal course of action in a Pokemon battle. In a battle, there are two different types of actions: use one of the four moves known by ...
2
votes
0answers
33 views

How to compare the weight matrices from different gradient descent runs?

I would like to compare the end results of gradient descent for runs which use different initial weights and different orders of training data. I know that index by index in the matrix will make no ...
0
votes
1answer
83 views

What is the expression for derivative of the signum function one should use in the BP training method

The back propagation learning method requires knowing of derivatives of activation functions. But what expression one should use for signum activation function $$ \mathrm{Signum}( x ) = \...
2
votes
1answer
104 views

Why convoloution neural net have to find filter values ?

I'm new to ML stuff and one of the thing that I don't understand about CNN, is that why CNN have to find the values of filter at convolution layer, why don' they use existing filters and only find the ...
7
votes
2answers
171 views

Clarification of the intuition behind backpropagation

I've been taking some time to try and understand the computations and mechanics of the machine learning algorithms I use in my day to day life. Studying the backpropagation literature on the CS231n ...
1
vote
1answer
198 views

Why don't neural networks get stuck in loops when they overshoot a backprop step?

In a normal feedforward network I wrote with linear activations I've noticed that after a while when the network has found a pretty viable solution to a problem it sometimes takes a step in the wrong ...
0
votes
1answer
299 views

How is the Cross-Entropy Cost Function back-propagated?

I've looked at a few threads about this but they've not been exactly what I'm after. When back-propagating the quadratic cost function, you first find the output error from $\delta_L = \...
5
votes
1answer
1k views

Transfer learning: How and why retrain only final layers of a network?

In this video, Prof. Andrew Ng says regarding transfer learning: Depending on how much data you have, you might just retrain the new layers of the network, or maybe you could retrain even more ...
0
votes
1answer
549 views

How does backpropagation differ from reverse-mode autodiff

Going through this book, I am familiar with the following: For each training instance the backpropagation algorithm first makes a prediction (forward pass), measures the error, then goes through ...
4
votes
1answer
699 views

What is the difference among stochastic, batch and mini-batch learning styles?

So far as I know, we have the following scenario: stochastic: The error is calculated for each sample s. So, we can calculate the gradients for s. And we can update the weights of the network ...
29
votes
4answers
17k views

Why is tanh almost always better than sigmoid as an activation function?

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$. The reason he gives is that the outputs using $tanh$ ...
2
votes
2answers
463 views

Can we use backpropagation to fit other models?

It appears that backpropagation is exclusively used to train neural network models. Why not use it to fit other models. For example - Taylor polynomials: $$ f(x) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)...
12
votes
1answer
1k views

Matrix form of backpropagation with batch normalization

Batch normalization has been credited with substantial performance improvements in deep neural nets. Plenty of material on the internet shows how to implement it on an activation-by-activation basis. ...
39
votes
1answer
21k views

How is softmax_cross_entropy_with_logits different from softmax_cross_entropy_with_logits_v2?

Specifically, I suppose I wonder about this statement: Future major versions of TensorFlow will allow gradients to flow into the labels input on backprop by default. Which is shown when I ...
10
votes
0answers
4k views

Backpropagation on a convolutional layer

Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually). First let me try ...
1
vote
2answers
236 views

Backpropagation gradients don't match approximated gradients

I am in the process of implementing back propagation into my image classification neural net. I am using this cost function with a sigmoid output layer and ReLU hidden layers. The neural net has 3 ...
0
votes
1answer
46 views

Inefficient Gradient Calculation in Neural Nets

I understood backprop: get gradient wrt a parameter (i.e. the partial derivative) using the chain rule. In the post http://www.offconvex.org/2016/12/20/backprop/ the authors say that the inefficient ...
0
votes
1answer
383 views

Backpropagation matrix multiply error Andrew Ng Machine Learning

In the Backpropagation algorithm video(Gradient Computation, week 5), he has taken an example neural network of 4 layers.(Input, 2 Hidden, Output). So, I had made my own example, I have taken <...
2
votes
0answers
340 views

Why aren't vanishing gradients for deep networks a problem?

This wikipedia article for Autoencoders states the following [07.30am UTC, 28th November 2017]: An autoencoder is often trained using one of the many variants of backpropagation (such as conjugate ...
2
votes
1answer
304 views

Deep NNs, backpropagation and error calculation [duplicate]

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
4
votes
1answer
460 views

Deep NNs, backpropagation and error calculation

I was following the backpropagation tutorial by Michael Nielsen on http://neuralnetworksanddeeplearning.com/chap2.html, one of the very few places where the backprop algorithm is nicely explained both ...
0
votes
1answer
596 views

back-propagation derivatives

In the 10th video of week3 of Ng course on Deep Neural Networks in coursera, there is a slide that i attached. Why he used elementwise product (vs normal matrix product) in this slide? Is it only for ...