Questions tagged [gradient-descent]
Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.
990
questions
0
votes
0
answers
37
views
Maximum Likelihood Estimation with Gradient Descent and Squarred Loss
My goal is to learn parameters $\mu$ and $\sigma$ of a univariate Gaussian distribution using gradient descent to validate my understanding of the algorithm by deriving all the formulas from scratch. ...
- 1
0
votes
0
answers
27
views
Should the target be standardized in gradient descent?
Suppose that we have a general loss function that depends on some parameters $w$ (e.g. neural network weights):
$$L_w =\frac{1}{N} \sum_i \ell(\hat{y}_i, y_i)$$
Is it beneficial to standardize the ...
- 541
0
votes
0
answers
25
views
Why is Stochastic Gradient Descent valid?
It seems unclear to me why SGD/minibatch GD works. I heard from someone that SGD works because "as commonly seen in stochastic optimization, the gradient step is an unbiased estimator of the true ...
- 173
1
vote
1
answer
26
views
Adding of Baseline parmter in derivation of Gradient Bandit Algorithm
In the derivation of the Gradient Bandit Algorithm in Chapter 2.8 of the Reinforcement Learning book by Sutton & Barto they introduce a introduce a baseline term $B_t$ and I can't seem to figure ...
- 121
1
vote
2
answers
82
views
ADALINE simple implementation with 2 features bug
I am reading Machine Learning with PyTorch and Ski-kit learn book by Sebastian Raschka
While plotting the decision boundary (a line in this case, since the number of features considered = 2) I can't ...
- 21
2
votes
1
answer
36
views
Do convergence rates for (convex) gradient descent apply when domain is (convex) subset of reals?
I have a convex multi-variate optimization problem where each variable lies on the domain $[x, \infty)$ for some positive number $x$. I know the problem has a unique finite solution in the domain, ...
- 133
1
vote
0
answers
20
views
SVRG vs full gradient descent
Stochastic gradient descent allows us to avoid the computation of full gradients at the expense of introducing a noise floor to convergence. To decrease this noise floor, SGD requires a decrease in ...
- 111
1
vote
0
answers
14
views
Problem with high learning rate in model training
In this lecture notes Fig. 6.5, it illustrates the effect to training loss by using different learning rates:
I don't understand why it seems a high enough non-divergent learning rate can converge to ...
- 413
0
votes
0
answers
11
views
Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?
Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
- 1
0
votes
1
answer
44
views
Question on the Partial Derivative of the Cross-Entropy Loss in SGD for Neural Networks
I'm currently learning about neural networks and stumbled upon a confusion related to the use of Stochastic Gradient Descent (SGD) in training. Specifically, I'm puzzled about the computation of the ...
John Title
0
votes
2
answers
136
views
Why do we maximize likelihood (sum of logs) and not simply maximize sum of probabilities? [duplicate]
In logistic regression we find the maximum likelihood estimator - $\max \prod_{i} p(y_i \mid x_i)$. Which in practice means maximizing the sum of log likelihoods. This makes sense, I understand MLE.
...
2
votes
1
answer
48
views
Computing gradient over all examples in gradient descent
I am studying about Gradient Descent and Stochastic Gradient Descent, and the text says that one of the advantages of sgd over gd is, that gd can be computationally expensive for large datasets. In ...
- 103
1
vote
0
answers
36
views
Group Lasso optimization
I read that, for the group lasso, to solve the zero subgradient equations, one approach involves keeping all block vectors fixed, denoted as $\{\hat\theta_k, k \ne j\}$, and then solving for $ \hat \...
- 261
0
votes
0
answers
13
views
Backpropagation gradients diffrence between my NN and Keras
I wrote a neural network from scratch and it works but I have a question there. My NN uses full batch and the gradients are therefore calculated with the entire matrix. Because I was interested, I ...
- 21
2
votes
0
answers
40
views
Solving a system of equalities using a neural network
Assume $P$ is a set of pairs $(x, y)$,
where both $x$ and $y$ are in $\mathbb{R}^n$.
Assume $P'$ is a subset of $P$.
I want to train a neural network
$N: \mathbb{R}^n \to \mathbb{R}^m$
such that, for ...
- 21
0
votes
0
answers
38
views
How does the chain-rule look for the gradient of a loss function?
When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like:
$...
- 1
0
votes
0
answers
10
views
Update rule with gradient of loss AND loss itself multiplied
I was reading a paper about Neural Holography (page 5, equation 4), where authors used simple stochastic gradient descent as optimizing method. There I have encountered following update rule:
, where ...
- 101
0
votes
1
answer
28
views
What loss function should I use to fit a distribution of points with a function with latent variables?
For simplicity I am going start with a toy example.
Lets suppose we have a set of $n$ points $\vec{Y}$ in the 2d space, distributed with the shape of the letter M. ...
- 101
0
votes
0
answers
16
views
Gradient descent loss increase [duplicate]
I am following the CS231n NN case study — a derivation of gradient descent for a simple network with a single hidden layer.
I have followed the rest of the tutorial and have confidence that the ...
- 709
0
votes
0
answers
26
views
Learning the gradient descent stepsize
I'm trying to apply reinforcement learning to learn the optimal stepsize in every gradient descent step. Currently I'm only considering simple quadratic problems of the form f(x)=x'Qx with Q diagonal ...
- 1
1
vote
0
answers
46
views
Neural ODE and Adjoint method
I am trying to understand the paper NeuralODE which is very interesting.
I get the general idea and the proof they give about the dynamics of the adjoint are fairly simple.
They define a network $f$, ...
0
votes
2
answers
205
views
A Simple Toy ML problem that surprisingly fails to converge (or even "try"!)
This is a much simplified network from a real problem that, to me, has a surprising INability to learn a simple task via backprop, ie, it can't overfit or learn at all. This simple version has come at ...
- 107
0
votes
0
answers
48
views
Subgradient of multi-class hinge loss
Let the multi-class hinge loss be given by:
$$ L(\Theta,(x,y)) = \max_{j \neq y} \Big[ 1+ \sum_{i=1}^{d} x_i(\Theta_{ij} - \Theta_{iy}) \Big]_{+} $$
$$ = \max_{j \neq y} \Big[ 1+ \langle x, \Theta_{\...
- 265
2
votes
0
answers
29
views
Implement Nesterov's acceleration for SVM
I am trying to implement Nestrov's acceleration gradient descent for SVM. The objective function I need to minimize is $$\frac{1}{2}\lVert Au-Bv\rVert_2^2$$ with constraints $\sum_{i}u_i=\sum_{j}v_j=1$...
0
votes
0
answers
33
views
Gradient descent residual
I've implemented the gradient descent method for finding roots of a system of nonlinear equations and I am wondering how the residual is determined? Is the residual simply the Euclidean norm (2-norm) ...
- 1
0
votes
0
answers
58
views
Best practices when training an xgboost model as part of a larger model?
I would like to train a model end-to-end that uses the output from an xgboost model as an input. I've successfully implemented full-batch gradient descent into my pipeline with jax, following this ...
1
vote
0
answers
24
views
Residuals in Online Gradient Descent
Does anyone know if online linear learning assumes white noise to the residuals?
My thought process is that serial correlation can arise due to the fact that the fit at time t uses the information ...
- 31
2
votes
0
answers
154
views
How to include parameter constraints in the Levenberg-Marquardt algorithm?
Every resource I found online doesn’t say what to do if the constraints aren’t satisfied. If my updated parameters is given by:
$$\theta_{k+1} = \theta_k - (JJ^T + \lambda I)^{-1}Jr, $$ where J is the ...
0
votes
0
answers
29
views
Why we use MSE in linear regression gradient descent rather than RSS? [duplicate]
Why in linear regression, when we use gradient descent, the cost function is MSE rather than RSS?
10
votes
2
answers
1k
views
Why go through the trouble of expectation maximization and not use gradient descent?
In expectation maximization first a lower bound of the likelihood is found and then a 2 step iterative algorithm kicks in where first we try to find the weights (the probability that a data point ...
- 2,613
1
vote
0
answers
56
views
Do common implementations of mini-batch gradient descent violate the i.i.d assumption needed for unbiased estimation?
When we perform mini-batch GD, we estimate the true gradient:
$$\nabla L = \frac{1}{N} \sum_i \nabla L_i$$
with:
$$\nabla_B L = \frac{1}{B} \sum_{i \in B} \nabla L_i$$
where $B$ is the batch size. ...
- 541
0
votes
0
answers
111
views
Is my custom loss function differentiable?
Consider the following loss function.
loss = ( ( torch.where(d > threshold, torch.sqrt(d), 0) * t ) + ( torch.where(d <= threshold, (1 - d), 0) * (1 - t) ) )
...
- 295
0
votes
0
answers
107
views
Convergence in Logistic Regression
Hey I'm taking a deeper dive into logistic regression. Specifically the following loss function with L2 regularization,
$$l(w)=\frac{1}{n}\sum_n \log(1+\exp(-y_i \cdot x_i^Tw))+\frac{\lambda}{2}||w||^...
- 1
1
vote
2
answers
64
views
How does feature scaling improve convergence in gradient descent?
Based on answers online, it appears that feature scaling helps to (1) ensure balanced step sizes and (2) make the cost function more symmetrical.
Why would an imbalanced step size be an issue, since a ...
- 11
1
vote
0
answers
33
views
Why I get spikes during training with vanilla gradient descent? [closed]
I developed my own NN toolbox, and it seems it works fine.
But I am not sure why I get these spikes in my loss during training:
I a training for a classification task of 2 inputs and 2 classes, ...
- 119
4
votes
2
answers
81
views
Is optimization without function evaluations (and with only gradient evaluations) possible?
I am implementing an unconstrained optimization algorithm using gradient descent. I am evaluating a cost function at a given point, evaluating the gradient at this point, and selecting the next ...
1
vote
0
answers
201
views
Understanding Backpropagation with Softmax and Quadratic Error
I'm trying to understand how to compute the derivative of the Softmax activation function in order to compute the gradient of the quadratic cost function w.r.t. the weights of the last layer.
...
2
votes
0
answers
40
views
How do we know if we are in a local or global minimum when using gradient descent?
I have been learning about Real Analysis and recently undergone a stats module on regressions etc.. We have come across gradient descent and I was curious about, if we have a complicated loss function,...
- 45
0
votes
0
answers
11
views
What's the point of using gradient descent for linear regression if you can calculate the coefficients directly using the least squares method? [duplicate]
Gradient descent involves significant computational effort, whereas the method of least squares enables direct and accurate calculation. Does gradient descent offer any advantages over least squares ...
- 11
1
vote
0
answers
34
views
Imputation method for missing values that are irrelevant
I have a data set $\mathbf X$, with around 20 predictors, which is a matrix of parameters of a surrogate model. For each observation $\mathbf i$ of $\mathbf X$, the surrogate model was trained to ...
- 153
0
votes
0
answers
128
views
Exploding coefficients in a Linear Regression Model with single input
This is my first post here, so tips are appreciated for future posts!
For homework in my CS course I wanted to build a very simple Linear Regression Model (although this was not explicitly required) ...
0
votes
2
answers
79
views
How does batch normalization enable larger learning rates (according to the original paper)?
I struggle to understand how batch normalization (BN) enables larger learning rates during gradient descent according to the original paper. I am aware that some of the explanations given in the ...
- 23
5
votes
2
answers
441
views
What is the meaning of "SGD scales the gradient uniformly in all directions"?
I'm really newbie about neural network and optimization. When I read the references, I found this journal Wang et al 2018.
The journal stated:
One disadvantage of SGD
is that it scales the gradient ...
- 127
0
votes
0
answers
107
views
XGBoost Algorithm: Impact of Scaling Instance Weights
Does scaling the instance weights impact the XGBoost ML Model training?
The instance weights determine the importance of each data point during the training process. The XGBoost ML Model uses instance ...
- 1
2
votes
1
answer
76
views
Exploding/Vanishing gradients deeper understanding
I'm trying to gain deper understanding of the logic behind vanishing and exploding gradients. Most sources I've come across explain the problem by saying that when the weights become too small, the ...
- 23
1
vote
0
answers
43
views
Minimizing the loss function seems counterintuitive
Loss functions I've known of always returns a positive value so that gradient descent can minimize them. Let's take a regression problem. Suppose my NN underestimate the result by x, then the loss ...
- 223
0
votes
1
answer
180
views
Why is my polynomial regression with gradient descent not overfitting?
I wanted to implement linear regression with gradient descent from scratch and demonstrate how you can overfit when using too many polynomials. Unfortunately my model does not really overfit the data. ...
- 107
0
votes
0
answers
75
views
How do you find the gradients of weights and biases in neural network during back propagation?
I have been trying to create a neural network from scratch.
I have been trying to calculate the gradients of the weights and biases of the neural network by watching videos and reading papers, but ...
1
vote
1
answer
558
views
What is the difference between a solver and an optimization algorithm?
In scikit-learn's LogisticRegression docs they write
This class implements regularized logistic regression using the ‘liblinear’ library, ‘newton-cg’, ‘sag’, ‘saga’ and ‘lbfgs’ solvers
Logistic ...
- 23
2
votes
1
answer
1k
views
Does normalizing/changing the scale of the target variable impact the shape of the loss function equation?
I was under the impression that changing the scale/normalizing the target variable in a regression task would not change the overall shape of the loss function equation but would simply translate/move ...
