Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
0
votes
1answer
23 views

Can we use Gradient Descent in the place of Ridge Regression in overfitting problem while doing linear regression problem?

What is the difference between Gradient Descent and Ridge regression? We use ridge regression for overfitting problem when the Mean Squared Error for test dataset is high. I think that we can use ...
2
votes
0answers
38 views
+50

Bayesian Regression- Expectation Maximization

In Bayesian regression, we have $y_i=x_i^{T}w+\epsilon_i$ where $w \sim \mathcal{N}(0,\alpha)$ and $\epsilon_i \sim \mathcal{N}(0,\frac{1}{\beta})$. Inference of $\alpha$ and $\beta$ is done by ...
2
votes
2answers
166 views

The difference between SGD and GD after use of backprop

This is a pretty basic question. So...backprop is an efficient algorithm for computing the gradients used by the optimizer to improve model parameters, no matter if SDG or something else. I get that. ...
0
votes
0answers
17 views

Dependency of the activation function in gradient descent calculations

I am working on linear classification script that uses gradient descent to do a binary classification of an object based on two features. I'm working with just a neuron. The output of the neuron uses ...
2
votes
0answers
51 views

Why fit on the gradient rather than minimize the loss directly when using Gradient Boosting?

In Friedman's paper on Gradient Boosting, he states the motivation for the gradient boosting algorithm is that it provides a framework of boosting for arbitrary loss functions. He then steps through a ...
0
votes
1answer
32 views

Rounding output of sigmoid for binary linear classifier

I am working on a linear classifier with expected output to be 1 for class A belonging and 0 for class B belonging. The output, in some occasions is nearly 0 (...
1
vote
0answers
15 views

A step of gradient descent is upper bounded by a function of gradient norm?

In Appendix A.2 of paper "Maximum a Posteriori Policy Optimisation"(Page 17), the authors say $f$ in Equation $(23)$ is simply a gradient descent update w.r.t. $\theta$ from the original $f$...
0
votes
0answers
9 views

What measure of heteroscedasticity is best for the use case of generating correlated data using gradient descent?

I am looking at generating datasets with correlated variables using gradient descent. There are already methods for generating correlated variables using Cholesky decomposition or eigenvalue ...
0
votes
0answers
20 views

Cant understand linear regression algorithm

I'm doing some machine learning stuff. And stumbled upon the machine learning regression algorithm. Here the derivatives are the ones from MSE: $$f(m,b) = \frac{1}{N} \sum_{i=1}^{n} (y_i - (mx_i + b))...
0
votes
0answers
7 views

Gradient descent in the space of distributions opposed to the ordinary gradient descent

In the conventional approach dynamics of the weights is simply a "slipping" along the hill to the bottom of the valley: $$ \frac{d \mathbf{W}}{dt} = -\nabla_{\mathbf{W}} L $$ In the mean ...
0
votes
0answers
13 views

Unstable Out-of-Sample Prediction with Gradient Boosting Trees

I have a continuous response, ranging between -100 and 100, but it's highly leptokurtic at 0. I also have a lot of predictors to use. After variable reduction and parameter tuning, the prediction of ...
0
votes
2answers
40 views

What does it mean if weights do not change but b does in gradient descent

Let say i have a learning line (y=wx+b). While training b value was constantly decreasing while there was no significant change on the w value during the subsequent iterations. In terms ...
2
votes
1answer
31 views

SGD is sensitive to feature scaling

I am taking a deep learning class and the class slides state one of SGD's problems as: "Gradient is scaled equally across all dimensions." Now what is meant by this is I believe, when we ...
1
vote
1answer
33 views

How to compute the gradient for a seperable nonlinear least squares problem?

Consider the case of non-linear least squares regression with one dependent variable $y_i$ and two independent variables $x_{i1}$ and $x_{i2}$ where the non-linear function is a linear function of two ...
6
votes
1answer
162 views

SGD for Gaussian Process estimation

Given a Gaussian process with kernel function $K_{\theta}$ depending on some hyperparameters $\theta$ and set of observations $\{(x_i,y_i)\}_{i=1}^n$, I want to choose $\theta$ to maximize the ...
2
votes
0answers
18 views

Gradient boosting: tree that fits the gradient of the custom loss function always uses squared loss?

With gradient boosting for regression, there are 2 loss functions, i.e: a custom loss function that we calculate the gradient for: $L(y_i,\hat{y_i})$ the loss function used by the tree that fits the ...
0
votes
0answers
21 views

Can a neural network still manage to converge, with slightly incorrect gradients?

In a network, we find gradients of the error function w.r.t each of the parameters used in the network. We then update the weights say, using vanilla Gradient Descent. If the computed gradients, do ...
1
vote
0answers
28 views

Gradient boosting doubts

I am new to the concept of Gradient Boosting and i have a few doubts related to it. It will be helpful if some one can explain them. 1) Gradient boosting is gradient descent in functional space As ...
1
vote
0answers
84 views

Convergence of gradient descent with ordinary least squares

I am currently trying to understand the numerical optimisation of the Ordinary Least Squares (OLS) approach to linear regression using gradient descent. So let us define the associated optimisation ...
0
votes
0answers
52 views

Negative Log Likelihood For Multiclass Logistic Regression

I have $P(\textbf{T | X},w_1,w_2,...,w_k) = \Pi_{n=1}^{N} \Pi_{k=1}^{K} P(C_k|x_n)^{t_{nk}} = \Pi_{n=1}^{N} \Pi_{k=1}^{K} y_{nk}^{t_{nk}}$ Where $\textbf{T}$ is N x K binary matrix of target variables ...
0
votes
1answer
53 views

Maximum Likelihood for Normal Distribution with Unknown Variance - Gradient Descent not working

Context The maximum likelihood estimators for a Normal distribution with unknown mean and unknown variance are $$ \widehat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i \qquad \text{and} \qquad \widehat{\sigma}^...
1
vote
0answers
13 views

how do i calculate (and Apply) Loss gradients with respect to the input (not the weights) of a CNN?

I have a trained generator, i would like to apply a loss function to the output and optimize the input (latent vector) using a gradient decent optimizer. i don't know how to calculate the gradients ...
0
votes
0answers
11 views

Understanding adding up of gradients for branched variables in chain rule of calculus in the context of neural network backpropagation

I was trying to understand how we can calculate gradients in back propagation in the context of neural network from here. It says following: The forward expression involves the variables $x,y$ ...
2
votes
0answers
23 views

Avoiding exploding gradients when forget gate is 1 and input/output gates are 0

Initially, the cell state equation was $C_t = C_{t-1} + i_t \odot \tanh(w_xx_t + w_hh_{t-1})$. Then to avoid exploding gradients, we added a forget gate such that $$C_t = f_t \odot C_{t-1} + i_t \odot ...
0
votes
1answer
81 views

How to derivate the following loss function?

How can I derivate the following optimization function? $$L=\sum_{u,i}(y_{u,i}-v_ix_u)^2+\lambda\left(\sum_i\|v_i\|_2^2+\sum_u\|x_u\|_2^2\right)$$ I just want to get the equations of the gradient ...
0
votes
0answers
7 views

can alternating optimization be performed in mini-batches

Just wondering if alternating minimization could be performed in mini-batches (just like we have gradient descent and its mini-batch version). Although I am perfectly fine with the full batch version ...
1
vote
0answers
32 views

How to choose initial $\theta$ (intercept and $\beta$s) in simple linear regression? [closed]

I have the sales of items from January 2013 to October 2015. I just want to predict the total sales for the next month. Just for the sake of learning, I would like to transform it into a multiple ...
2
votes
1answer
85 views

What does vanishing gradient problem exactly mean? [duplicate]

I'm currently doing some stuff with ML, and I created a LSTM model to recognize activities such as walking or running. I have read that LSTM has advantages over traditional RNNs due to addressing the ...
2
votes
1answer
27 views

Small, simple neural network test problem?

I beginning to learn about neural networks and how to train them. I've been reading about using gradient descent for training. Most books go straight to the backpropogation algorithm for computing $\...
5
votes
0answers
46 views

Local optima in high-dimensional optimization

I remember a theorem along the lines of In higher dimensional optimization problems, you are less likely to get stuck in local optima, because the more dimensions you have, the more likely you are to ...
0
votes
0answers
24 views

Running accelerated gradient descent on $\prod_{i=1}^{n}\alpha \beta y_{i}^{\beta - 1}exp(-\alpha y_{i}^\beta)$

Running accelerated gradient descent on $\prod_{i=1}^{n}\alpha \beta y_{i}^{\beta - 1}exp(-\alpha y_{i}^\beta)$ I have this code for running AGD on the above function in MatLab ...
0
votes
0answers
26 views

Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?

The title is one of the special things in Progressive GAN, a paper of the NVIDIA team. By using this method, they introduced that Our approach ensures that the dynamic range, and thus the learning ...
2
votes
1answer
12 views

can we perform sub-gradient updates in mini-batches

We are already aware that in case the data is quite bulky, mini-batch gradient descent based approaches may be applied. These approaches load a mini-batch of data, compute the loss on this batch, and ...
0
votes
0answers
16 views

Stochastic gradient decent using local computation

I am new to the ML field especially the federated learning topics, and I am trying to use this formula to estimate local computation in FL. In this paper they provide the equation for local ...
0
votes
0answers
15 views

What does the loss function adjust in gradient boosting

I get that a gradient boosting algorithm adds weak learners together in order to fit some data. I also get that each subsequent learner after the first one fits the residuals of the current model. ...
0
votes
0answers
18 views

Backpropagation With Sigmoid Output Function Question

I am deriving a Weight update for a simple toy network with a Sigmoid Output Layer. I need some help double checking my math to make sure I did it correctly. I am using Cross-Entropy Loss as my Loss ...
2
votes
1answer
50 views

Why doesn't the optimizer just look for stationary points of the loss function?

I want to have a better understanding of the weight-optimization process. I understand the optimizer(e.g., gradient descent) looks for the direction in which to move the parameters to minimize the ...
1
vote
0answers
23 views

Science practice: Where to introduce approximations?

In my work, I am using an algorithm which relies on estimates of the gradient of the log-posterior at a collection of Monte Carlo samples. Since this gradient is not available in closed form, I must ...
1
vote
1answer
16 views

Should gradient descent based algorithms return the best score?

While studying gradient descent based optimizers I have been wondering if they should return the value after the last update or the value that provides the smallest loss. Pros: returning the best ...
1
vote
1answer
98 views

vanishing gradient and gradient zero

There is a well known problem vanishing gradient in BackPropagation training of ...
0
votes
0answers
22 views

why are squares of gradient used in adaptive learning rates techniques like rmsprop? [duplicate]

I have already had a look at this question Please elaborate the answer and would further like to add, what if we don't use squared derivatives.
0
votes
0answers
47 views

Score function of multi-variate normal distribution

There is an existing question about the score function of multivariate-normal distributions (i.e. the gradient of the log-pdf), but it is somewhat specific to the problem of the OP, and I am ...
0
votes
0answers
21 views

Conceptual explanation of taking derivative of sigmoid function in neural network

In a neural network, we have a bunch of inputs and corresponding weights + a bias which are represented by a multivariable equation. Now we squash this whole equation with a sigmoid function. How ...
0
votes
0answers
21 views

Deriving vectorized form of linear regression

We first have the weights of a D dimensional vector $w$ and a D dimensional predictor vector $x$, which are all indexed by $j$. There are $N$ observations, all D dimensional. $t$ is our targets, i.e, ...
3
votes
1answer
119 views

How Gradient Descent is used for classification with Decision Trees?

I'm not able to see how do we use gradient descent to minimize the loss of binary classification with decision tree. What I understood is that we first have a model (decision tree) that try to predict ...
2
votes
1answer
95 views

Understanding mini-batch gradient descent

I would like to understand the steps of mini-batch gradient descent for training a neural network. My train data $(X,y)$ has dimension $(k \times n)$ and $(1 \times n)$, where $k$ is the number of the ...
1
vote
0answers
92 views

Bias in Gradient Descent (GD) and Stochastic GD (SGD)

Let $\theta$ be weight parameters and assume the loss function to be $L_N(\theta)=\frac{1}{N}\sum_i f(\theta; x_i,y_i)$. Assume a mini batch loss function with a batch of size $M$ and denote the loss ...
0
votes
0answers
13 views

Non - convergence of loss function in Neural Networks with a step activation function [duplicate]

In the below lecture by Professor Patrick H. Winston on Neural Networks: https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-034-artificial-intelligence-fall-2010/lecture-videos/...
2
votes
0answers
52 views

Gradient Descent diverges when input data changes

I'm trying to use the gradient descent to solve the following min problem: $$\min_R \sum_{j=m}^K|| X_j - RX_{j-i}||_2^2, X\in R^{n \times K}$$ For simplicity, assume $m=i+1,\text{ and } i = 1$. That ...
4
votes
3answers
266 views

Basic preconditioned gradient descent example

I'm exploring preconditioned gradient descent using a similar toy problem described in the first part of Lecture 8: Accelerating SGD with preconditioning and adaptive learning rates. I have the ...

1
2 3 4 5
17