Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

Gradient Boosting vs Gradient Descent

The section 10.10.2 of ESL claims that the difference between gradient boosting and steepest descent is that The tree components $t_{m} = T(x_{1};\theta_{m}),...,T(x_{N};\theta_{m})$ are not ...
2
votes
1answer
39 views

Gradient based optimization of step function w.r.t number of steps

I am trying to optimize the parameter b in the following simple function using gradient descent in PyTorch: $$ y = \frac{\lfloor{xb} \rfloor + 0.5}{b} $$ x is in $[0,1]$ and b is continuous and in $[5,...
0
votes
1answer
30 views

Meaning of a varaible for calculating the partial derviative of MSE cost function

The equation to find the partial derivative of a cost function with respect to a parameter θj is given in the book 'Hands on Machine Learning with scikit-learn, ...
0
votes
0answers
13 views

Relationship between number of NN parameters and required training iterations

How does the number of parameters influence the training time in terms of iterations? I'm aware that the training takes more hours the more parameters you have, but would like to know whether the ...
1
vote
1answer
21 views

Understanding step in proof of GAN algorithm convergence, involving convexity

I am reading the original paper on GANs, https://arxiv.org/abs/1406.2661. The proof of proposition 2, on the convergence of the gradient descent algorithm reads Consider $V(G, D) = U(p_g, D)$ as a ...
1
vote
0answers
9 views

Q Learning Target and Prediciton

The following formula is unclear to me That Q(s,a) is my prediction make sense to me. But why is (r+ Vopt(s')) my target. s' is the successor. So my prediction for Q(s,a) should be near to Q(s',a)? ...
6
votes
2answers
79 views

What to do when your training and testing data have different distributions

I am training a XGBoost regression model for predicting number of applications and the range of the target variable in train and test data set is different. For e.g: In Train data : Minimum ...
3
votes
0answers
27 views

Question about minimising empirical loss by gradient descent

Say we wanted to learn $f_{\theta}(\pmb{x})=y$, with a loss function $L(f_{\theta}(\pmb{x}),y)$. We often want to choose $\theta$ which minimises the empirical loss, as the exact loss isn't available ...
1
vote
2answers
45 views

Intuition About Gradient Descent Convergence

I know that gradient descent takes steps towards a minimum, but I am having trouble coming up with intuitions about when it will converge. For example, on any given convex function is gradient descent ...
1
vote
0answers
22 views

Stochastic Gradient Descent - Least Squares [closed]

I am not sure if I implemented the SGD in a proper way since in calculations it gives way to big error even on the training set. Can you help me to figure out where I made a mistake? Here $D$ is the ...
0
votes
0answers
36 views

What is an initial consistent estimator and how do I find one?

When maximizing a likelihood function $L(\psi)$, the gradient-based optimization procedure is generally $$ \tag{5.1} \hat{\psi}_{r+1} = \hat{\psi}_{r} + \left| I^{*}(\hat{\psi}_{r}) \right|^{-1} D \...
2
votes
2answers
43 views

Are the training samples shuffled in minibatch gradient descent? [duplicate]

Lets say that I have 3,200 observations that I want to use for training a neural network model, and I want to set the batch size to 32. The number of minibatches used in every epoch for updating the ...
0
votes
0answers
9 views

How can I calculate derivative of cross-entropy with mini-batch gradient descend?

How can we calculate the derivative of cross-entropy with mini-batch gradient descend? As we know, we have this formula for stochastic gradient descend: $$ {w^k} = {w^k} - \alpha*({P_y}_k - 1)*{x^i} \ ...
0
votes
0answers
7 views

Is there a better convex algorithm better than Dual Gradient Descent

Is there a better convex algorithm better than Dual Gradient Descent https://jonathan-hui.medium.com/rl-dual-gradient-descent-fac524c1f049
1
vote
1answer
23 views

Why are vanishing gradients an issue in a minimization problem?

From reading various articles and questions posted on this site, I understand how sometimes layers of a deep neural network may not learn at the same rates, and some (especially earlier) layers may ...
3
votes
1answer
34 views

Is Convexity necessary to use Gradient descent?

I was reading and I saw that convexity is sufficient for using GD to minimize functions. Would it be also necessary?
1
vote
0answers
8 views

What is the range of learning rate which makes gradient descent converge and not converge for this loss?

Range in terms of x and y As you might be aware that gradient descent uses alpha = learning rate W* = W -alpha*(dL/dW) x,y are constant
2
votes
1answer
19 views

Visualize tangent plane for mean squared error loss function

I would really appreciate your help on a rather simple issue that I just can't solve on my own. I'd like to visualize gradient descent for a simple linear regression problem with two parameters $\...
1
vote
0answers
50 views

How to simply understand gradient boosting on ranking problem?

I am reading Chris Burge's paper about LambdaRank, LambdaMART for learning to rank. We only need to compute the lambda, which is relevant to gradients, and use it to update model parameters, no need ...
3
votes
1answer
32 views

Do we use maximum likelihood or cross entropy Loss for training skip-gram model?

In the skip gram model, maximising the likelihood of the context words given the middle word is equivalent to minimising the objective function $J(\theta)$, where $$J(\theta) = -\frac{1}{T}\sum_{t=1}^...
0
votes
0answers
10 views

How to analyze sensitivity of an optimizer to initial parameter values?

Is there a standard way to analyze sensitivity of an optimizer (like gradient descent) to initial value of the parameters? Some objective functions may have diverse local optima and by starting from ...
0
votes
0answers
12 views

Questions about mirror descent

I have 2 questions about the fundamental properties of mirror descent. Assume $D \subset R^n$ is an open and convex set, $X \subset R^n$ is a convex set, assume $D \cap X \ne \emptyset$, $X \subset \...
0
votes
1answer
20 views

I have doubts about descending gradient and backpropagation?

i am a beginner in this from ai and i am learning about gradient descent and its update rule Is it true that the same gradient is applied to each weight of the network at each step? that is, for ...
1
vote
1answer
15 views

Mini/Batch Gradient Descent, sum of gradients

Summing all gradients in one mini/batch descent is a special methode or an obligation to get the correct gradient. Does making median/mean/maxes/... methodes to have different 'global' gradient from ...
0
votes
0answers
37 views

Coding gradient descent from scratch - how are fitness functions incorporated into output layer error calculation?

For a project I am currently working on, I'm attempting to implement machine learning for a neural network using backpropagation and gradient descent from scratch. For much of my implementation, I ...
2
votes
1answer
38 views

What is meant by Expressiveness in neural network?

While studying Batch normalization, I came across the parameter sigma and beta in the output. And all the information said that they are added in order to retain the "expressive power of the ...
2
votes
1answer
23 views

Relationship between value of epoch and mini-batch and the total number of calculations

I've read this regarding the difference between epoch and mini-batch. To clarify: With an ...
0
votes
0answers
29 views

Is convexity actually necessary for gradient descent?

I understand that the problem that when you are optimizing something using gradient descent, the algorithm might get stuck in a local optimum that isn't global. Otherwise, there are non-convex ...
0
votes
0answers
23 views

How do shared weight vectors work for CRF?

I am going through two materials regarding Conditional Random Fields. The first one is this (referred to as [1]) material by Charles Sutton and Andrew McCallum and the second one is this (referred to ...
0
votes
0answers
18 views

When is a high learning rate for Stochastic Gradient Descent a good thing?

I was always under the impression that SGD needed a lower learning rate than optimizers like Adam, because it was stochastic and more likely to make training diverge with higher learning rates. I ...
0
votes
0answers
34 views

Best way to deal with non-continuously differentiable error functions in machine learning

Suppose you have the following set of $n=9$ numbers x = [1, 2, 3, 4, 5, 6, 7, 8, 9] and the corresponding y values [10, 2.5, 1.1, 0.6, 0.4, 0.2, 0.15, 0.12] which visualized look like this: I'd like ...
0
votes
1answer
18 views

Vectorised Implementation of SVM Gradient

I am trying to implement the SVM loss and gradient. The loss is given as $$L(w) = \sum_{i=1}^N max\{1-y_iw^tx_i, 0 \} + \lambda ||w^2||_2^2$$ I believe that for the loss, this is a good implementation;...
0
votes
0answers
7 views

Difference between O(epsilon) and O(iterations)

Often in algorithms like gradient-descent (and variants), we see the algorithmic complexity either in terms of O(1/$[epsilon]$), while on the other hand some papers show it in terms of number of ...
0
votes
1answer
37 views

Can we use Gradient Descent in the place of Ridge Regression in overfitting problem while doing linear regression problem?

What is the difference between Gradient Descent and Ridge regression? We use ridge regression for overfitting problem when the Mean Squared Error for test dataset is high. I think that we can use ...
3
votes
1answer
110 views

Bayesian Regression- Expectation Maximization

In Bayesian regression, we have $y_i=x_i^{T}w+\epsilon_i$ where $w \sim \mathcal{N}(0,\alpha)$ and $\epsilon_i \sim \mathcal{N}(0,\frac{1}{\beta})$. Inference of $\alpha$ and $\beta$ is done by ...
2
votes
2answers
210 views

The difference between SGD and GD after use of backprop

This is a pretty basic question. So...backprop is an efficient algorithm for computing the gradients used by the optimizer to improve model parameters, no matter if SDG or something else. I get that. ...
0
votes
0answers
17 views

Dependency of the activation function in gradient descent calculations

I am working on linear classification script that uses gradient descent to do a binary classification of an object based on two features. I'm working with just a neuron. The output of the neuron uses ...
2
votes
0answers
67 views

Why fit on the gradient rather than minimize the loss directly when using Gradient Boosting?

In Friedman's paper on Gradient Boosting, he states the motivation for the gradient boosting algorithm is that it provides a framework of boosting for arbitrary loss functions. He then steps through a ...
0
votes
1answer
35 views

Rounding output of sigmoid for binary linear classifier

I am working on a linear classifier with expected output to be 1 for class A belonging and 0 for class B belonging. The output, in some occasions is nearly 0 (...
1
vote
0answers
15 views

A step of gradient descent is upper bounded by a function of gradient norm?

In Appendix A.2 of paper "Maximum a Posteriori Policy Optimisation"(Page 17), the authors say $f$ in Equation $(23)$ is simply a gradient descent update w.r.t. $\theta$ from the original $f$...
0
votes
0answers
10 views

What measure of heteroscedasticity is best for the use case of generating correlated data using gradient descent?

I am looking at generating datasets with correlated variables using gradient descent. There are already methods for generating correlated variables using Cholesky decomposition or eigenvalue ...
0
votes
0answers
23 views

Cant understand linear regression algorithm

I'm doing some machine learning stuff. And stumbled upon the machine learning regression algorithm. Here the derivatives are the ones from MSE: $$f(m,b) = \frac{1}{N} \sum_{i=1}^{n} (y_i - (mx_i + b))...
0
votes
0answers
10 views

Gradient descent in the space of distributions opposed to the ordinary gradient descent

In the conventional approach dynamics of the weights is simply a "slipping" along the hill to the bottom of the valley: $$ \frac{d \mathbf{W}}{dt} = -\nabla_{\mathbf{W}} L $$ In the mean ...
0
votes
0answers
18 views

Unstable Out-of-Sample Prediction with Gradient Boosting Trees

I have a continuous response, ranging between -100 and 100, but it's highly leptokurtic at 0. I also have a lot of predictors to use. After variable reduction and parameter tuning, the prediction of ...
0
votes
2answers
45 views

What does it mean if weights do not change but b does in gradient descent

Let say i have a learning line (y=wx+b). While training b value was constantly decreasing while there was no significant change on the w value during the subsequent iterations. In terms ...
2
votes
1answer
59 views

SGD is sensitive to feature scaling

I am taking a deep learning class and the class slides state one of SGD's problems as: "Gradient is scaled equally across all dimensions." Now what is meant by this is I believe, when we ...
1
vote
1answer
44 views

How to compute the gradient for a seperable nonlinear least squares problem?

Consider the case of non-linear least squares regression with one dependent variable $y_i$ and two independent variables $x_{i1}$ and $x_{i2}$ where the non-linear function is a linear function of two ...
5
votes
1answer
253 views

SGD for Gaussian Process estimation

Given a Gaussian process with kernel function $K_{\theta}$ depending on some hyperparameters $\theta$ and set of observations $\{(x_i,y_i)\}_{i=1}^n$, I want to choose $\theta$ to maximize the ...
2
votes
0answers
48 views

Gradient boosting: tree that fits the gradient of the custom loss function always uses squared loss?

With gradient boosting for regression, there are 2 loss functions, i.e: a custom loss function that we calculate the gradient for: $L(y_i,\hat{y_i})$ the loss function used by the tree that fits the ...
0
votes
0answers
21 views

Can a neural network still manage to converge, with slightly incorrect gradients?

In a network, we find gradients of the error function w.r.t each of the parameters used in the network. We then update the weights say, using vanilla Gradient Descent. If the computed gradients, do ...

1
2 3 4 5
17