Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

990 questions
Filter by
Sorted by
Tagged with
37 views

Maximum Likelihood Estimation with Gradient Descent and Squarred Loss

My goal is to learn parameters $\mu$ and $\sigma$ of a univariate Gaussian distribution using gradient descent to validate my understanding of the algorithm by deriving all the formulas from scratch. ...
27 views

Should the target be standardized in gradient descent?

Suppose that we have a general loss function that depends on some parameters $w$ (e.g. neural network weights): $$L_w =\frac{1}{N} \sum_i \ell(\hat{y}_i, y_i)$$ Is it beneficial to standardize the ...
• 541
25 views

Why is Stochastic Gradient Descent valid?

It seems unclear to me why SGD/minibatch GD works. I heard from someone that SGD works because "as commonly seen in stochastic optimization, the gradient step is an unbiased estimator of the true ...
1 vote
26 views

In the derivation of the Gradient Bandit Algorithm in Chapter 2.8 of the Reinforcement Learning book by Sutton & Barto they introduce a introduce a baseline term $B_t$ and I can't seem to figure ...
• 121
1 vote
82 views

ADALINE simple implementation with 2 features bug

I am reading Machine Learning with PyTorch and Ski-kit learn book by Sebastian Raschka While plotting the decision boundary (a line in this case, since the number of features considered = 2) I can't ...
• 21
36 views

Do convergence rates for (convex) gradient descent apply when domain is (convex) subset of reals?

I have a convex multi-variate optimization problem where each variable lies on the domain $[x, \infty)$ for some positive number $x$. I know the problem has a unique finite solution in the domain, ...
• 133
1 vote
20 views

Stochastic gradient descent allows us to avoid the computation of full gradients at the expense of introducing a noise floor to convergence. To decrease this noise floor, SGD requires a decrease in ...
• 111
1 vote
14 views

Problem with high learning rate in model training

In this lecture notes Fig. 6.5, it illustrates the effect to training loss by using different learning rates: I don't understand why it seems a high enough non-divergent learning rate can converge to ...
• 413
11 views

Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?

Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
44 views

Question on the Partial Derivative of the Cross-Entropy Loss in SGD for Neural Networks

I'm currently learning about neural networks and stumbled upon a confusion related to the use of Stochastic Gradient Descent (SGD) in training. Specifically, I'm puzzled about the computation of the ...
136 views

Why do we maximize likelihood (sum of logs) and not simply maximize sum of probabilities? [duplicate]

In logistic regression we find the maximum likelihood estimator - $\max \prod_{i} p(y_i \mid x_i)$. Which in practice means maximizing the sum of log likelihoods. This makes sense, I understand MLE. ...
48 views

I am studying about Gradient Descent and Stochastic Gradient Descent, and the text says that one of the advantages of sgd over gd is, that gd can be computationally expensive for large datasets. In ...
• 103
1 vote
36 views

10 views

Update rule with gradient of loss AND loss itself multiplied

I was reading a paper about Neural Holography (page 5, equation 4), where authors used simple stochastic gradient descent as optimizing method. There I have encountered following update rule: , where ...
• 101
28 views

What loss function should I use to fit a distribution of points with a function with latent variables?

For simplicity I am going start with a toy example. Lets suppose we have a set of $n$ points $\vec{Y}$ in the 2d space, distributed with the shape of the letter M. ...
• 101
16 views

I am following the CS231n NN case study — a derivation of gradient descent for a simple network with a single hidden layer. I have followed the rest of the tutorial and have confidence that the ...
• 709
26 views

I'm trying to apply reinforcement learning to learn the optimal stepsize in every gradient descent step. Currently I'm only considering simple quadratic problems of the form f(x)=x'Qx with Q diagonal ...
1 vote
46 views

I am trying to understand the paper NeuralODE which is very interesting. I get the general idea and the proof they give about the dynamics of the adjoint are fairly simple. They define a network $f$, ...
205 views

A Simple Toy ML problem that surprisingly fails to converge (or even "try"!)

This is a much simplified network from a real problem that, to me, has a surprising INability to learn a simple task via backprop, ie, it can't overfit or learn at all. This simple version has come at ...
• 107
48 views

• 1
1 vote
64 views

How does feature scaling improve convergence in gradient descent?

Based on answers online, it appears that feature scaling helps to (1) ensure balanced step sizes and (2) make the cost function more symmetrical. Why would an imbalanced step size be an issue, since a ...
• 11
1 vote
33 views

Why I get spikes during training with vanilla gradient descent? [closed]

I developed my own NN toolbox, and it seems it works fine. But I am not sure why I get these spikes in my loss during training: I a training for a classification task of 2 inputs and 2 classes, ...
• 119
81 views

Is optimization without function evaluations (and with only gradient evaluations) possible?

I am implementing an unconstrained optimization algorithm using gradient descent. I am evaluating a cost function at a given point, evaluating the gradient at this point, and selecting the next ...
1 vote
201 views

Understanding Backpropagation with Softmax and Quadratic Error

I'm trying to understand how to compute the derivative of the Softmax activation function in order to compute the gradient of the quadratic cost function w.r.t. the weights of the last layer. ...
40 views

How do we know if we are in a local or global minimum when using gradient descent?

I have been learning about Real Analysis and recently undergone a stats module on regressions etc.. We have come across gradient descent and I was curious about, if we have a complicated loss function,...
• 45
11 views

What's the point of using gradient descent for linear regression if you can calculate the coefficients directly using the least squares method? [duplicate]

Gradient descent involves significant computational effort, whereas the method of least squares enables direct and accurate calculation. Does gradient descent offer any advantages over least squares ...
• 11
1 vote
34 views

Imputation method for missing values that are irrelevant

I have a data set $\mathbf X$, with around 20 predictors, which is a matrix of parameters of a surrogate model. For each observation $\mathbf i$ of $\mathbf X$, the surrogate model was trained to ...
• 153
128 views

Exploding coefficients in a Linear Regression Model with single input

This is my first post here, so tips are appreciated for future posts! For homework in my CS course I wanted to build a very simple Linear Regression Model (although this was not explicitly required) ...
79 views

How does batch normalization enable larger learning rates (according to the original paper)?

I struggle to understand how batch normalization (BN) enables larger learning rates during gradient descent according to the original paper. I am aware that some of the explanations given in the ...
441 views

What is the meaning of "SGD scales the gradient uniformly in all directions"?

I'm really newbie about neural network and optimization. When I read the references, I found this journal Wang et al 2018. The journal stated: One disadvantage of SGD is that it scales the gradient ...
• 127
107 views

XGBoost Algorithm: Impact of Scaling Instance Weights

Does scaling the instance weights impact the XGBoost ML Model training? The instance weights determine the importance of each data point during the training process. The XGBoost ML Model uses instance ...
76 views

I'm trying to gain deper understanding of the logic behind vanishing and exploding gradients. Most sources I've come across explain the problem by saying that when the weights become too small, the ...
1 vote
43 views

Minimizing the loss function seems counterintuitive

Loss functions I've known of always returns a positive value so that gradient descent can minimize them. Let's take a regression problem. Suppose my NN underestimate the result by x, then the loss ...
• 223
180 views

Why is my polynomial regression with gradient descent not overfitting?

I wanted to implement linear regression with gradient descent from scratch and demonstrate how you can overfit when using too many polynomials. Unfortunately my model does not really overfit the data. ...
• 107
75 views

How do you find the gradients of weights and biases in neural network during back propagation?

I have been trying to create a neural network from scratch. I have been trying to calculate the gradients of the weights and biases of the neural network by watching videos and reading papers, but ...
1 vote