Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there ...

1
vote
0answers
32 views

Linear least squares algorithms

I have stumbled across these two questions and accepted answers: (1) Do we need gradient descent to find the coefficients of a linear regression model? (2) Why use gradient descent for linear ...
2
votes
1answer
24 views

Backpropagation wrong? Doesn't it update dependent variables in hidden layer

In a multi layer perceptron or feedforward neural network, isn't backpropagation updating weights of the middle layers that are dependent variables? So for a particular hidden layer, it calculates all ...
0
votes
0answers
9 views

Back-propagated gradients vs Weight gradients? [on hold]

I am trying to reproduce “Understanding the difficulty of training deep ffnn” using PyTorch and extend the same analysis to more scenarios. I have both theoretical and practical questions regarding ...
1
vote
0answers
21 views

Gradient Descent and the No-Free-Lunch Theorem

I'm doing a presentation on the No-Free-Lunch theorem, since I've found that the best way to learn about a topic is to try and teach it...In order to get an idea what I'm talking about, I started ...
1
vote
0answers
23 views

Stochastic gradient descent vs mini-batch gradient descent

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
1
vote
2answers
44 views

Why is the second derivative required for newton's method for back-propagation?

I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely. I have seen this same question, and the widely accepted answer ...
0
votes
0answers
9 views

Logistic regression subgradients with different regularizers

Consider the following regularized logistic regression problem: $$\textbf{w}^* = \min_{\textbf{w} \in {\rm I\!R}^p} \mathcal{L}(\textbf{w}) $$ where $$ \mathcal{L}(\textbf{w}) = \frac{1}{n} \sum_{...
0
votes
0answers
10 views

scipy L-BFGS-B optimizer with different step size per dimension [migrated]

How can I adjust the optimizer to use a different step size for each DOF? When I print the parameters the step size seems to the the same per dimension. Any other alternative optimizer that can ...
0
votes
0answers
10 views

Neural network backpropagation by gradient descent is better than conjugate gradient?

My understanding is that the conjugate gradient method is faster than gradient descent because it does less zig zags while descending. How come the state of the art papers I see all use gradient ...
1
vote
0answers
38 views

Possible questions on testing on hypothesis in A/B testing for a position of a data scientist (EDITED)

What are some possible machine learning and statistics questions on the subject of A/B testing for a position of a data scientist in Ad Tech industry working in retargeting? I'm interviewing for such ...
0
votes
1answer
24 views

SGD and quantile regression

It is my understanding that the quantile loss is not differentiable (at 0) so base gradient descent cannot be used. However, Vowpal Wabbit which is an SGD-based learner very much includes quantile ...
0
votes
0answers
7 views

Can adaptive learning rate method be used for dropout regularization?

if the neurons are deactivated randomly for each forward pass during an iteration, Can adaptive learning rate method for neural network such as RMSprop be used for the case of dropout regularization?
1
vote
0answers
21 views

Batch learning in digits recognition (MNIST database) [duplicate]

While working my way through M. Nielsen's "Neural networks and deep learning", I decided to try out some presumably silly things to really understand why they won't work and/or why it's not a good ...
2
votes
2answers
43 views

Why gradient descent of log probabilities in Alpha Go?

In the original AlphaGo paper, it is stated that the policy network is trained with the following gradient: I don't understand why this gradient makes sense. Why do we want to move the parameters ...
0
votes
0answers
13 views

Exploding Gradient Problem and Increasing Net Error even with Batch Normalization?

I've implemented just about everything there is such as standardizing the data set, implementing batch normalization but the problem is that the more training iterations i go through, the net error ...
1
vote
0answers
27 views

Influence of RMSE versus MSE on Convergence of Gradient Descent

I am working in TensorFlow and was wondering if I choosing an MSE loss function would cause different convergence behaviour when compared to a RMSE loss function. The square root will influence the ...
1
vote
0answers
30 views

Why the sign is *plus* in neural network [closed]

REFERENCE GITHUB GIST I wanted to implement Neural Network with Numpy in Python. Then I have two question. The first one is about the sign ...
0
votes
0answers
25 views

Calculating minimum of a function using Gradient Descent

I need to calculate the minimum of function : f(x) = (x − 3)2 , starting at x = 0 and with α = 1/3, by applying gradient descent. Could someone please help me here how to go about it? Found no clear ...
9
votes
4answers
1k views

Gradient descent optimization

I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a ...
0
votes
0answers
10 views

Gradient for deep neural net in matrix notation?

I am used to the notation for gradient that is written explicitly in the individual parameters. What you get is a sum of the derivatives w.r.t each parameter. But is there a way to write this down ...
0
votes
0answers
26 views

In what sense is momentum “optimal” for optimisation of neural networks?

This article states about the use of momentum in gradient descent for neural networks: A lower bound, courtesy of Nesterov [5], states that momentum is, in a certain very narrow and technical sense,...
0
votes
0answers
8 views

What should be my initial values of coefficients while iterating over gradient descent for logistic regression?

I have understood how to get the cost function of logistic regression. Now I want to iterate and perform a gradient descent on the function. Should I choose {0} as my initial values for coefficients?
3
votes
0answers
28 views

Gradient Descent in Metric Learning for Kernel Regression (MLKR)

I am currently studying the Metric Learning for Kernel Regression (MLKR) algorithm (http://proceedings.mlr.press/v2/weinberger07a/weinberger07a.pdf). Let $\{(x_{1}, y_{1}), ..., (x_{N}, y_{N})\}$ ...
0
votes
0answers
9 views

Simple fix to make nonlinear activation functions more trainable?

It is generally considered that sigmoid type activation functions are less trainable than ReLu's because the gradient vanishes at high and low parameter values. However I don't see why this cannot be ...
0
votes
1answer
14 views

Gradient descent parameter calculation seems wrong? [closed]

I try to calculate the first epoch of a gradient descent.I only have one feature X;X=[1,2,3]; Y is [-1,-4,-7], so the function is 2-3x;so theta0=2, theta1=-3; I try to predict them with a learning ...
0
votes
1answer
39 views

Why is gradient descent and it's variants used instead of BFGS and L-BFGS for training neural nets? [duplicate]

My understanding is that BFGS and L-BFGS solve the same type of optimization problems as GD and it's variants. Why is GD the go to algorithm for training neural networks?
1
vote
0answers
28 views

Gradient descent decreasing loss

Is the following statement true: Gradient descent is guaranteed to always decrease a loss function. I know that if the loss function is convex, then each iteration of gradient descent will result in ...
1
vote
0answers
11 views

3 Layer neural network (feedforward and backpropagation) - Desired error not necessarily achieved due to precision loss

I am trying to implement a 3 layer neural network with feedforward and backpropagation. I have tested my cost function and it is working fine. My gradient function also seems ok. but when I try to ...
0
votes
0answers
42 views

Real-coded GAs (RCGAs) Vs Steepest Descent method

I'm just learning Genetic Algorithm and Optimization methods, What I want to ask is is there any advantages and disadvantages between Real-Coded GAs (RCGAs) compared to the Steepest Descent method?
0
votes
0answers
13 views

gradient with respect to totally different things in neural networks vs gradient boosting?

I'm confused about the usage of gradients in NN vs. GBM. Is is correct to say that the gradient is with respect to totally different things? my understanding is that: in NN (I'm following ...
1
vote
0answers
89 views

How the Hessian matrix is used in optimization if you can't invert it

I've seen quite a lot of work to do with approximating the Hessian such as the Hessian Vector Product but I'm not entirely sure how knowing the Hessian helps us evaluate the gradient step to take. ...
0
votes
0answers
33 views

Calculating the Goldilocks learning rate

In Google's machine learning crash course, they have an experiment of how to find the Goldilocks learning rate. Looks like the curve is from this function: f(x)=1.875x^2-7.675x+7.86 AFAIK the ideal ...
1
vote
0answers
19 views

Higher Order of Vectorization in Backpropagation in Neural Network

I am learning a machine learning class online from Stanford, namely CS 229. There is one section about deep learning and back-propagation in deep learning. The network looks like: The forward ...
0
votes
1answer
134 views

stochastic gradient descent of ridge regression when regularization parameter is very big

As we know, the gradient of ridge regression is: $$ g = \frac{\partial L}{\partial \theta} = -X_i^T(y_i-X_i\theta)+2\lambda\theta $$ where $X_i$ is the $i$th training sample. The update of $\theta$ is ...
0
votes
1answer
39 views

Finding maximum likelihood solution of a continuous state HMM

The likelihood of a hidden Markov model (HMM) for states $x_0, \dots, x_N$ and observations $y_1, \dots, y_N$ can be written as $$ L = f(x_0) \prod_{i=1}^N f(y_i | x_i) f(x_i | x_{i-1} )$$ where we ...
22
votes
5answers
2k views

For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

Given a convex cost function, using SGD for optimization, we will have a gradient (vector) at a certain point during the optimization process. My question is, given the point on the convex, does the ...
1
vote
1answer
84 views

Why doesn't feature standardization make SGD with momentum redundant?

In the paper An overview of gradient descent optimization algorithms, the author discusses the Momentum algorithm: SGD has trouble navigating ravines, i.e. areas where the surface curves much ...
1
vote
1answer
96 views

understanding natural policy gradient

I'm reading this paper on Natural Policy Gradient https://papers.nips.cc/paper/2073-a-natural-policy-gradient.pdf and have some questions regarding how it works. I'm coming at this from an ML ...
3
votes
0answers
118 views

Why does Nesterov momentum not improve the rate of convergence in the stochastic gradient case?

I have been reading Deep Learning book by Ian Goodfellow, where they wrote in chapter 8 (section 8.3.3) that Nesterov momentum does not improve the rate of convergence in stochastic gradient case. ...
1
vote
0answers
19 views

Long repetitive output after changing vocabulary in seq2seq model

I trained a neural question generation model, which produces sensible questions for the vocabulary that they distributed with the paper. I wanted to run the model on a different set of word embeddings ...
0
votes
0answers
39 views

R constrOptim: Do I need to calculate the gradient with respect to variables that only appear in constraints?

I have a problem of the form $$\min h(w_v, w_o) = w_v'\Sigma w_v-w_o'\Sigma w_o$$ subject to $$w_v=w_L-w_T+\Delta_v^+-\Delta_v^-$$ $$w_o=w_L-w_T+\Delta_o^+-\Delta_o^-$$ $$\Delta_v^+ + \Delta_{v....
0
votes
0answers
51 views

Model converges with Adam (batch size=128/64) but not with Adam (batch size=32) or SGD (any batch size)

I've built a network for a binary classification task, where a "file" of documents is mapped to a single outcome. My architecture is - ...
2
votes
0answers
21 views

How to compare the weight matrices from different gradient descent runs?

I would like to compare the end results of gradient descent for runs which use different initial weights and different orders of training data. I know that index by index in the matrix will make no ...
1
vote
0answers
30 views

RNN: Why is my loss declining although my gradients are zero?

I am using a recurrent neural network in Tensorflow with an AdaGradOptimizer. For understanding of RNNs I set the gradients to zero manually like this: ...
1
vote
1answer
143 views

How can change in cost function be positive?

In chapter 1 of Nielsen's Neural Networks and Deep Learning it says To make gradient descent work correctly, we need to choose the learning rate η to be small enough that Equation (9) is a good ...
1
vote
1answer
70 views

Simple examples for cases in which gradient descent diverges

I am trying to acquire an intuition and understanding of cases in which gradient descent (GD) diverges. I have found some questions (1, 2, 3) in which people tried to minimize a popular cost function ...
0
votes
2answers
31 views

Is a loss function computed after each step of gradient descent or after a whole epoch?

In neural networks with mini-batch or stochastic gradient descent, is a loss function computed after each step of gradient descent or after a whole epoch?
1
vote
0answers
18 views

Does TRPO train for multiple steps on the same data?

I understand mainly how Trust Region Policy Optimisation (TRPO) works, in terms of the trust region constraining the gradient updates from straying too far from the 'old' policy. Does this mean that ...
0
votes
0answers
28 views

How To Choose Step Size (Learning Rate) in Batch Gradient Descent?

I'm practicing machine learning in python and trying to implement batch gradient descent algorithm by myself. Mathematically algorith is defined as follows: $\theta_j = \theta_j + \alpha \sum_{i=0}^{...
3
votes
1answer
146 views

Gradient descent in SVM

I am studying about SVM now. Then I came across the problem. The dual optimization problem is as follows: \begin{align*} &\max_\alpha~~~~~ W(\alpha) = \sum_{i=1}^{n} \alpha_i -\frac{1}{2}\...