# Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

632 questions
9 views

### How to get precise answer from stochastic gradient descent

I have a convex optimization problem in few variables and I have an unbiased estimator of the gradient without having the ability to evaluate the function itself. I want to do gradient descent but the ...
13 views

### Large Feature Values for Gradient Descent

Recently, I work on a linear regression model of my project. I have 200 samples, each of which has only one feature, to train my model. When I try to apply ...
8 views

### How can I implement this Robust PCA equation in a more efficient way?

I recently learned in class the Principle Component Analysis method aims to approximate a matrix X to a multiplication of two matrices Z*W. If X is a n x d matrix, Z is a n x k matrix and W is a k x d ...
300 views

### Is stochastic gradient descent pseudo-stochastic?

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times. ...
15 views

### word2vec gradient update clarification

I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice. The gradient in ...
12 views

### Will gradient descent prefer stronger signals?

Let's say we have a linear regression problem: $$\mathbf{y} = \mathbf{X}_1\mathbf{\beta}_1 + \mathbf{X}_2\mathbf{\beta}_2 + \mathbf{\epsilon}$$ where $\mathbf{X}_1$ and $\mathbf{X}_2$ are sampled ...
12 views

12 views

### Confusion about the derivation of the TD-Learning update rule

I am currently trying to understand the paper "Learning to Predict by the Methods of Temporal Differences" by Sutton. I am stuck with the following step: (From "Learning to Predict by the Methods of ...
17 views

### Implicit regularization in Linear models

Regarding Linear Neural Networks models with unique finite root loss function, without an explicit regularization, I am struggling to prove that in the case of overparmeterized models (i.e. $N<d$), ...
107 views

7 views

### How gradient decent training will affect if we use feature-crosses or high-order polynomials?

Considering Multivariate linear regression. We use feature scaling + mean normalization(feature transformation) on our features to keep them on the same scale. If we don't do that then our contour ...