Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

0
votes
1answer
22 views

Back propagation is done with each batch in a convolutional net, but is it also done with the validation set?

It's my understanding that the weights are updated in a convolutional neural network with each evaluation of a batch. But when the training data has been processed and it comes to predicting ...
0
votes
0answers
31 views

Order of subtraction in gradient of squared-error

Consider a regression problem with the following loss function: $$ L(\theta) = (\hat{y}(\theta) - y)^2 $$ when doing gradient descent, we do $$ \theta \gets \theta - \alpha \nabla_\theta L(\theta) $$...
1
vote
1answer
26 views

What is the computational cost of gradient descent vs linear regression?

I know the computational costs for the closed form of linear regression is $O(n^3)$, but I can't find a similar cost comparison to gradient descent. There are some similar questions here with people "...
0
votes
0answers
15 views

What is main difference between AdamOptimizer and GradientDescentOptimizer? [duplicate]

I'm just dealing with a simple linear regression algorithm: ...
0
votes
0answers
24 views

Why are analytical solutions possible for some optimization problems but not possible for others?

For example, an analytical solution is possible for linear regression (i.e. the normal equations) but it is not possible for logistic regression but logistic regression can be optimized with gradient ...
2
votes
1answer
24 views

Question about the gradient of weight normalization

In Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, they define the weight vector as $$ \mathbf w={g\over\Vert\mathbf v\Vert}\mathbf v $$ Then they ...
1
vote
0answers
27 views

Gradient descent versus fixed point iteration

Fixed-point iteration Say I have the iteration $$x^{(k+1)} \leftarrow x^{(k)} + \alpha f(x^{(k)})$$ to find $x^\ast$, the root of $f$, i.e. $f(x^\ast)=0$, where $f:(a,b) \to \mathbb{R}$, $\exists ...
2
votes
0answers
18 views

Fisher information matrix and gradients

I'm a math Ph.D. without formal training in statistics. Quite a few papers on normalization methods in deep learning mention the Fisher information matrix and how it's related to the Riemannian metric ...
1
vote
0answers
26 views

Momentum updates average of g, Adagrad also of g^2 - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
0
votes
1answer
27 views

What stops gradient descent from finding the largest error? [duplicate]

If a gradient points towards a max or a min what stops gradient descent from maximizing error instead of minimizing it? Is it the nature of the update step that makes this process one way?
1
vote
0answers
26 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
1answer
32 views

Are there are alternatives to gradient update rule?

Most optimization techniques (that I'm aware of) for non-linear cost functions that are commonly implemented rely on linearly updating a variable iteratively until a minimum is reached or a condition ...
0
votes
2answers
34 views

why is number of epochs set as external parameter?

I am confused by the very notion of epochs in neural networks (as well as number of trees in gradient boosting). Gradient descent method (as most optimization algorithms) keep going until the loss ...
2
votes
1answer
26 views

Do fully connected layers in the middle of a network impede optimization?

I submitted a paper that uses an auto-encoder network with several convolutional layers in both the encoder and the decoder and a fully connected layer (FCL) in between. Besides the FCL being useful ...
0
votes
0answers
12 views

How to quantify the effect of the inputs and the effect of the labels on our regression model?

Let us imagine in a regression setting where our responses are D dimensional vectors, and so are our inputs. Let's say I train a regression model (e.g, Neural network). When I update my weights on a ...
0
votes
1answer
24 views

Gradient descent with no analytical function f(x)

I find myself in the position of wanting to minimize a numerical function f(a,b) with respect to a and b for which I do not have a the analytical form f(a,b). All I have is f(a,b) for many values of a ...
1
vote
1answer
36 views

Does online data augmentation make sense?

Data augmentation is popularly done online as that is how it is typically implemented and suggested in neural network frameworks like Keras and TensorFlow. I have also seen it described in e.g. the ...
0
votes
0answers
11 views

How to get precise answer from stochastic gradient descent

I have a convex optimization problem in few variables and I have an unbiased estimator of the gradient without having the ability to evaluate the function itself. I want to do gradient descent but the ...
2
votes
0answers
14 views

Large Feature Values for Gradient Descent

Recently, I work on a linear regression model of my project. I have 200 samples, each of which has only one feature, to train my model. When I try to apply ...
1
vote
0answers
9 views

How can I implement this Robust PCA equation in a more efficient way?

I recently learned in class the Principle Component Analysis method aims to approximate a matrix X to a multiplication of two matrices Z*W. If X is a n x d matrix, Z is a n x k matrix and W is a k x d ...
4
votes
1answer
312 views

Is stochastic gradient descent pseudo-stochastic?

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times. ...
0
votes
0answers
27 views

word2vec gradient update clarification

I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice. The gradient in ...
0
votes
0answers
12 views

Will gradient descent prefer stronger signals?

Let's say we have a linear regression problem: $$ \mathbf{y} = \mathbf{X}_1\mathbf{\beta}_1 + \mathbf{X}_2\mathbf{\beta}_2 + \mathbf{\epsilon} $$ where $\mathbf{X}_1$ and $\mathbf{X}_2$ are sampled ...
0
votes
0answers
14 views

Gradient Descent method for Wolfe Duality Hyperplane Optimization

I am troubling myself by learning how to build an optimal hyperplane for a separable case using the $\texttt{iris}$ data in R. The function I am trying to maximize w.r.t $\alpha_{i}$ is: $$ L_D = \...
1
vote
0answers
47 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
0
votes
3answers
57 views

Weights not converging while cost function has converged in neural networks [closed]

I'm talking in an ideal scenario where a validation set isn't used. Without validation, as many epochs as possible are calculated. Training stops and finishes only when the loss function is minimized ...
1
vote
0answers
26 views

How to plot the gradient descent of a RNN model built using keras? [closed]

I'm exploring how an LSTM solves the problem of vanishing gradients. I have created a simple LSTM model on keras. I know that model.fit() returns a history object that stores model loss and accuracy ...
0
votes
1answer
26 views

Gradient boosting understanding of residual picture

I recently looking at the Gradient boosting using following blog https://medium.com/mlreview/gradient-boosting-from-scratch-1e317ae4587d I try to understand the picture but I need some help For ...
1
vote
1answer
59 views

Intuition behind computing gradient for a model

I'm trying to understand gradient computation. The basic formula I found looks like so: ...
1
vote
0answers
27 views

What is the difference between the validation loss on a regression task and the mean squared error?

The validation loss on regression task using mean squared error loss function is different from the mean squared error value directly calculated on the validation set. What is the difference between ...
5
votes
1answer
457 views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
1
vote
1answer
29 views

Feedforward networks: methods to avoid two neurons in the same layer learning the same weights and biases?

There are many questions on this site which have to do with "what happens when two neurons have the same weights/biases" and I am not asking about that. However, it is occasionally the case that a ...
1
vote
0answers
9 views

Nested CV with Online Learning

I have a time series binary classification dataset. I am implementing an online learning Logistic Regression algorithm in Sklearn and am cross validating with Sklearn's TimeSeriesSplit method. I am ...
0
votes
1answer
132 views

Loss during minibatch gradient descent

I have minibatch gradient descent code in Tensorflow for function approximation, but I am unsure when to calculate the loss. First, I create batches for x and y data. Then, I shuffle both these ...
1
vote
0answers
30 views

RMSProp Squared Gradients

In the RMSProp algorithm (And similar algorithms) that are used in Machine Learning in the subject of Adaptive Learning Rates, the squares of the gradients are used in the algorithm step. Is there ...
2
votes
1answer
27 views

Learning Rate impact on model building time

I wanted to know that does learning rate impact the model building time in case of Gradient Boosted Trees. I do understand that increasing the number of trees have an impact( more the trees, more the ...
1
vote
0answers
154 views

Effects of class imbalance on nn batch training

Say I have a binary classification task, where the positive class (1) is only 1% of the whole data set. Intuitively I can understand why this could be bad for the classifier as the model may learn ...
1
vote
1answer
63 views

Can you implement Replay Buffers for Reinforcement Learning when most experiences give zero reward?

Specifically, for a deep deterministic policy gradient, DDPG, to expedite the learning speed, it's recommended to use a Replay Buffer What if the reward is only given at a terminal state? Or, most of ...
2
votes
0answers
27 views

Learning Manifolds using Gradient Descent

I have a feedforward neural network $F(W): \mathbb R^d \rightarrow \mathbb R^k$ with $Relu$ activation, $m$ neurones per layer, $L$ layers and softmax on the output layer. $W$ denotes the weight ...
1
vote
0answers
53 views

Gradient descent expression help

In the section 2 of this paper. I didn't understand the following steps in the proof. If $x$ is channel input, $w$ channel weight vector and $\hat{w} = w/$ $ ||w||_2$ then for batchnorm ($BN$) we ...
0
votes
1answer
38 views

What is the rationale behind clipping gradients in binary neural networks?

In binarized neural networks, where activations are confined to -1/+1 values using the sign function, derivative of the sign function is estimated with a straight-through estimator. However, when I ...
0
votes
0answers
51 views

What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & ...
2
votes
1answer
181 views

Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
1
vote
2answers
33 views

In linear regression, is finding the minimum of the parameters the same as gradient descent?

I'm taking a course on ML and have just began. Given a loss function, $$L = \frac{1}{N}\sum^N_{n=1}(t_n - w_0 + w_1x_n)^2$$ I am confused between the difference of using gradient descent (and maybe ...
1
vote
1answer
49 views

Specifics on weight update calculation in stochastic gradient descent

I understand the difference between batch and stochastic gradient descent as follows: let's say there are only two samples and two features. Let the function used to estimate $y^{(i)}$ be: $$h_{\theta}...
0
votes
1answer
13 views

Confusion about the derivation of the TD-Learning update rule

I am currently trying to understand the paper "Learning to Predict by the Methods of Temporal Differences" by Sutton. I am stuck with the following step: (From "Learning to Predict by the Methods of ...
0
votes
0answers
18 views

Implicit regularization in Linear models

Regarding Linear Neural Networks models with unique finite root loss function, without an explicit regularization, I am struggling to prove that in the case of overparmeterized models (i.e. $N<d$), ...
1
vote
1answer
240 views

Derivation of Perceptron weight update formula

I've started out studying Machine Learning and am currently reading up about how a single perceptron works. From the wikipedia page, my understanding is as follows: suppose we have an input sample $\...
1
vote
1answer
28 views

Differences between “in-bag” and “out-of-bag” empirical risks in the R package “mboost”

currently I am using the mboost R-package to estimate some additive models. When using the function gamboost(), you can control the hyper-parameters for boosting by using the option boost_control(). ...
1
vote
0answers
9 views

Can factor analysis be fit with gradient-based methods?

I know you can fit factor analysis using EM, but can you use gradient-based methods? If so, a reference would be great; otherwise, why not?