Skip to main content

Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
1 vote
0 answers
10 views

Problem with high learning rate in model training

In this lecture notes Fig. 6.5, it illustrates the effect to training loss by using different learning rates: I don't understand why it seems a high enough non-divergent learning rate can converge to ...
Sam's user avatar
  • 373
0 votes
0 answers
11 views

Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?

Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
Kevin's user avatar
  • 1
0 votes
1 answer
26 views

Question on the Partial Derivative of the Cross-Entropy Loss in SGD for Neural Networks

I'm currently learning about neural networks and stumbled upon a confusion related to the use of Stochastic Gradient Descent (SGD) in training. Specifically, I'm puzzled about the computation of the ...
user avatar
0 votes
2 answers
115 views

Why do we maximize likelihood (sum of logs) and not simply maximize sum of probabilities? [duplicate]

In logistic regression we find the maximum likelihood estimator - $\max \prod_{i} p(y_i \mid x_i)$. Which in practice means maximizing the sum of log likelihoods. This makes sense, I understand MLE. ...
Amnon Attali's user avatar
2 votes
1 answer
44 views

Computing gradient over all examples in gradient descent

I am studying about Gradient Descent and Stochastic Gradient Descent, and the text says that one of the advantages of sgd over gd is, that gd can be computationally expensive for large datasets. In ...
WalaWizon's user avatar
  • 103
1 vote
0 answers
29 views

Group Lasso optimization

I read that, for the group lasso, to solve the zero subgradient equations, one approach involves keeping all block vectors fixed, denoted as $\{\hat\theta_k, k \ne j\}$, and then solving for $ \hat \...
Jenny's user avatar
  • 261
0 votes
0 answers
13 views

Backpropagation gradients diffrence between my NN and Keras

I wrote a neural network from scratch and it works but I have a question there. My NN uses full batch and the gradients are therefore calculated with the entire matrix. Because I was interested, I ...
Apro9991's user avatar
2 votes
0 answers
39 views

Solving a system of equalities using a neural network

Assume $P$ is a set of pairs $(x, y)$, where both $x$ and $y$ are in $\mathbb{R}^n$. Assume $P'$ is a subset of $P$. I want to train a neural network $N: \mathbb{R}^n \to \mathbb{R}^m$ such that, for ...
Mahyar's user avatar
  • 21
0 votes
0 answers
35 views

How does the chain-rule look for the gradient of a loss function?

When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like: $...
ZenPyro's user avatar
0 votes
0 answers
10 views

Update rule with gradient of loss AND loss itself multiplied

I was reading a paper about Neural Holography (page 5, equation 4), where authors used simple stochastic gradient descent as optimizing method. There I have encountered following update rule: , where ...
FirePapaya's user avatar
0 votes
1 answer
17 views

What loss function should I use to fit a distribution of points with a function with latent variables?

For simplicity I am going start with a toy example. Lets suppose we have a set of $n$ points $\vec{Y}$ in the 2d space, distributed with the shape of the letter M. ...
Iván's user avatar
  • 101
0 votes
0 answers
16 views

Gradient descent loss increase [duplicate]

I am following the CS231n NN case study — a derivation of gradient descent for a simple network with a single hidden layer. I have followed the rest of the tutorial and have confidence that the ...
Henry's user avatar
  • 709
0 votes
0 answers
22 views

Learning the gradient descent stepsize

I'm trying to apply reinforcement learning to learn the optimal stepsize in every gradient descent step. Currently I'm only considering simple quadratic problems of the form f(x)=x'Qx with Q diagonal ...
CodeGuy's user avatar
1 vote
0 answers
32 views

Neural ODE and Adjoint method

I am trying to understand the paper NeuralODE which is very interesting. I get the general idea and the proof they give about the dynamics of the adjoint are fairly simple. They define a network $f$, ...
Julien Séveno-Piltant's user avatar
-1 votes
2 answers
199 views

A Simple Toy ML problem that surprisingly fails to converge (or even "try"!)

This is a much simplified network from a real problem that, to me, has a surprising INability to learn a simple task via backprop, ie, it can't overfit or learn at all. This simple version has come at ...
Josh.F's user avatar
  • 97
0 votes
0 answers
40 views

Subgradient of multi-class hinge loss

Let the multi-class hinge loss be given by: $$ L(\Theta,(x,y)) = \max_{j \neq y} \Big[ 1+ \sum_{i=1}^{d} x_i(\Theta_{ij} - \Theta_{iy}) \Big]_{+} $$ $$ = \max_{j \neq y} \Big[ 1+ \langle x, \Theta_{\...
Oskar's user avatar
  • 245
2 votes
0 answers
25 views

Implement Nesterov's acceleration for SVM

I am trying to implement Nestrov's acceleration gradient descent for SVM. The objective function I need to minimize is $$\frac{1}{2}\lVert Au-Bv\rVert_2^2$$ with constraints $\sum_{i}u_i=\sum_{j}v_j=1$...
struggleinmath's user avatar
0 votes
0 answers
28 views

Gradient descent residual

I've implemented the gradient descent method for finding roots of a system of nonlinear equations and I am wondering how the residual is determined? Is the residual simply the Euclidean norm (2-norm) ...
blov's user avatar
  • 1
0 votes
0 answers
43 views

Best practices when training an xgboost model as part of a larger model?

I would like to train a model end-to-end that uses the output from an xgboost model as an input. I've successfully implemented full-batch gradient descent into my pipeline with jax, following this ...
ironicoxidant's user avatar
1 vote
0 answers
22 views

Residuals in Online Gradient Descent

Does anyone know if online linear learning assumes white noise to the residuals? My thought process is that serial correlation can arise due to the fact that the fit at time t uses the information ...
sebHan1234's user avatar
2 votes
0 answers
96 views

How to include parameter constraints in the Levenberg-Marquardt algorithm?

Every resource I found online doesn’t say what to do if the constraints aren’t satisfied. If my updated parameters is given by: $$\theta_{k+1} = \theta_k - (JJ^T + \lambda I)^{-1}Jr, $$ where J is the ...
THATS MY QUANT MY QUANTITATIVE's user avatar
0 votes
0 answers
29 views

Why we use MSE in linear regression gradient descent rather than RSS? [duplicate]

Why in linear regression, when we use gradient descent, the cost function is MSE rather than RSS?
krm fordev's user avatar
11 votes
2 answers
1k views

Why go through the trouble of expectation maximization and not use gradient descent?

In expectation maximization first a lower bound of the likelihood is found and then a 2 step iterative algorithm kicks in where first we try to find the weights (the probability that a data point ...
figs_and_nuts's user avatar
1 vote
0 answers
35 views

Do common implementations of mini-batch gradient descent violate the i.i.d assumption needed for unbiased estimation?

When we perform mini-batch GD, we estimate the true gradient: $$\nabla L = \frac{1}{N} \sum_i \nabla L_i$$ with: $$\nabla_B L = \frac{1}{B} \sum_{i \in B} \nabla L_i$$ where $B$ is the batch size. ...
ado sar's user avatar
  • 471
0 votes
0 answers
97 views

Is my custom loss function differentiable?

Consider the following loss function. loss = ( ( torch.where(d > threshold, torch.sqrt(d), 0) * t ) + ( torch.where(d <= threshold, (1 - d), 0) * (1 - t) ) ) ...
Adel's user avatar
  • 295
0 votes
0 answers
96 views

Convergence in Logistic Regression

Hey I'm taking a deeper dive into logistic regression. Specifically the following loss function with L2 regularization, $$l(w)=\frac{1}{n}\sum_n \log(1+\exp(-y_i \cdot x_i^Tw))+\frac{\lambda}{2}||w||^...
zzz's user avatar
  • 1
1 vote
2 answers
55 views

How does feature scaling improve convergence in gradient descent?

Based on answers online, it appears that feature scaling helps to (1) ensure balanced step sizes and (2) make the cost function more symmetrical. Why would an imbalanced step size be an issue, since a ...
Michael's user avatar
  • 11
1 vote
0 answers
32 views

Why I get spikes during training with vanilla gradient descent? [closed]

I developed my own NN toolbox, and it seems it works fine. But I am not sure why I get these spikes in my loss during training: I a training for a classification task of 2 inputs and 2 classes, ...
z_tjona's user avatar
  • 119
4 votes
2 answers
72 views

Is optimization without function evaluations (and with only gradient evaluations) possible?

I am implementing an unconstrained optimization algorithm using gradient descent. I am evaluating a cost function at a given point, evaluating the gradient at this point, and selecting the next ...
Joaquin Rapela's user avatar
0 votes
0 answers
43 views

Custom loss function optimized for correlation for gradient boosted trees algorithms like XGBRegressor or LGBMRegressor

I have a rather classical tabular data prediction problem and more or less successfully using XGBRegressor or LGBMRegressor both using MSE as their loss function. A slight deviation from the standard ...
Josef Švenda's user avatar
1 vote
0 answers
156 views

Understanding Backpropagation with Softmax and Quadratic Error

I'm trying to understand how to compute the derivative of the Softmax activation function in order to compute the gradient of the quadratic cost function w.r.t. the weights of the last layer. ...
Dario Ranieri's user avatar
2 votes
0 answers
39 views

How do we know if we are in a local or global minimum when using gradient descent?

I have been learning about Real Analysis and recently undergone a stats module on regressions etc.. We have come across gradient descent and I was curious about, if we have a complicated loss function,...
ILE2091's user avatar
  • 45
0 votes
0 answers
11 views

What's the point of using gradient descent for linear regression if you can calculate the coefficients directly using the least squares method? [duplicate]

Gradient descent involves significant computational effort, whereas the method of least squares enables direct and accurate calculation. Does gradient descent offer any advantages over least squares ...
Dawid's user avatar
  • 1
1 vote
0 answers
34 views

Imputation method for missing values that are irrelevant

I have a data set $\mathbf X$, with around 20 predictors, which is a matrix of parameters of a surrogate model. For each observation $\mathbf i$ of $\mathbf X$, the surrogate model was trained to ...
Florent H's user avatar
  • 153
0 votes
0 answers
110 views

Exploding coefficients in a Linear Regression Model with single input

This is my first post here, so tips are appreciated for future posts! For homework in my CS course I wanted to build a very simple Linear Regression Model (although this was not explicitly required) ...
Florian Ott's user avatar
0 votes
2 answers
76 views

How does batch normalization enable larger learning rates (according to the original paper)?

I struggle to understand how batch normalization (BN) enables larger learning rates during gradient descent according to the original paper. I am aware that some of the explanations given in the ...
Cipollino's user avatar
0 votes
0 answers
61 views

How to decide learning rate, iterations in ADVI with adam?

I am Automatic Differentiation Variational Inference for bayesian inference using PyMC. I am using ADAM optimizer learning rate = 0.0005, number of monte carlo objects = 2, 20,000 iterations and full ...
newbie's user avatar
  • 225
5 votes
2 answers
426 views

What is the meaning of "SGD scales the gradient uniformly in all directions"?

I'm really newbie about neural network and optimization. When I read the references, I found this journal Wang et al 2018. The journal stated: One disadvantage of SGD is that it scales the gradient ...
andryan86's user avatar
  • 127
0 votes
0 answers
82 views

XGBoost Algorithm: Impact of Scaling Instance Weights

Does scaling the instance weights impact the XGBoost ML Model training? The instance weights determine the importance of each data point during the training process. The XGBoost ML Model uses instance ...
cugurm's user avatar
  • 1
2 votes
1 answer
70 views

Exploding/Vanishing gradients deeper understanding

I'm trying to gain deper understanding of the logic behind vanishing and exploding gradients. Most sources I've come across explain the problem by saying that when the weights become too small, the ...
Cipollino's user avatar
1 vote
0 answers
43 views

Minimizing the loss function seems counterintuitive

Loss functions I've known of always returns a positive value so that gradient descent can minimize them. Let's take a regression problem. Suppose my NN underestimate the result by x, then the loss ...
Minh Triet's user avatar
1 vote
1 answer
149 views

Why is my polynomial regression with gradient descent not overfitting?

I wanted to implement linear regression with gradient descent from scratch and demonstrate how you can overfit when using too many polynomials. Unfortunately my model does not really overfit the data. ...
burton030's user avatar
  • 117
0 votes
0 answers
71 views

How do you find the gradients of weights and biases in neural network during back propagation?

I have been trying to create a neural network from scratch. I have been trying to calculate the gradients of the weights and biases of the neural network by watching videos and reading papers, but ...
ManOnTheMoon's user avatar
1 vote
1 answer
430 views

What is the difference between a solver and an optimization algorithm?

In scikit-learn's LogisticRegression docs they write This class implements regularized logistic regression using the ‘liblinear’ library, ‘newton-cg’, ‘sag’, ‘saga’ and ‘lbfgs’ solvers Logistic ...
statnoob's user avatar
2 votes
1 answer
958 views

Does normalizing/changing the scale of the target variable impact the shape of the loss function equation?

I was under the impression that changing the scale/normalizing the target variable in a regression task would not change the overall shape of the loss function equation but would simply translate/move ...
Kiran Manicka's user avatar
1 vote
0 answers
22 views

Could someone help me interpret the data that I have gathered thus far?

I am trying to train a SVM model for my statistical learning course. The problem is a binary image classification problem (wildfire, nowildfire). This is the rigorous amount of testing that I have ...
Ahsan Yousaf's user avatar
0 votes
0 answers
121 views

Target binning in regression

I would like to find a predictive density for target variable via multi-class classification. Suppose we are given a set of features $\mathbf X$ and continuous target $\mathbf y$. Replace each $y$ ...
vladkkkkk's user avatar
  • 691
1 vote
0 answers
50 views

Neural network parameters dependency vs gradient descent [duplicate]

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we ...
Glue's user avatar
  • 485
0 votes
1 answer
43 views

Why do the error derivatives become small if we start with a large learning rate?

In these slides from Hinton (https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf) there is this statement: I don't understand why "The error derivatives for the hidden units ...
Nick's user avatar
  • 1
0 votes
0 answers
67 views

My loss has a non-differentiable point

I had to design a loss function max(0,x). It's not differentiable at x=0. In order to train it with gradient descent, what should I do? I have learned that subgradient can be used instead, so does it ...
firstforst's user avatar

1
2 3 4 5
20