Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
1
vote
0answers
25 views

How to choose initial $\theta$ (intercept and $\beta$s) in simple linear regression? [closed]

I have the sales of items from January 2013 to October 2015. I just want to predict the total sales for the next month. Just for the sake of learning, I would like to transform it into a multiple ...
2
votes
1answer
26 views

What does vanishing gradient problem exactly mean? [duplicate]

I'm currently doing some stuff with ML, and I created a LSTM model to recognize activities such as walking or running. I have read that LSTM has advantages over traditional RNNs due to addressing the ...
1
vote
1answer
22 views

Small, simple neural network test problem?

I beginning to learn about neural networks and how to train them. I've been reading about using gradient descent for training. Most books go straight to the backpropogation algorithm for computing $\...
5
votes
0answers
42 views

Local optima in high-dimensional optimization

I remember a theorem along the lines of In higher dimensional optimization problems, you are less likely to get stuck in local optima, because the more dimensions you have, the more likely you are to ...
0
votes
0answers
23 views

Running accelerated gradient descent on $\prod_{i=1}^{n}\alpha \beta y_{i}^{\beta - 1}exp(-\alpha y_{i}^\beta)$

Running accelerated gradient descent on $\prod_{i=1}^{n}\alpha \beta y_{i}^{\beta - 1}exp(-\alpha y_{i}^\beta)$ I have this code for running AGD on the above function in MatLab ...
0
votes
0answers
22 views

Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?

The title is one of the special things in Progressive GAN, a paper of the NVIDIA team. By using this method, they introduced that Our approach ensures that the dynamic range, and thus the learning ...
2
votes
1answer
10 views

can we perform sub-gradient updates in mini-batches

We are already aware that in case the data is quite bulky, mini-batch gradient descent based approaches may be applied. These approaches load a mini-batch of data, compute the loss on this batch, and ...
0
votes
0answers
16 views

Stochastic gradient decent using local computation

I am new to the ML field especially the federated learning topics, and I am trying to use this formula to estimate local computation in FL. In this paper they provide the equation for local ...
0
votes
0answers
9 views

What does the loss function adjust in gradient boosting

I get that a gradient boosting algorithm adds weak learners together in order to fit some data. I also get that each subsequent learner after the first one fits the residuals of the current model. ...
0
votes
0answers
14 views

Backpropagation With Sigmoid Output Function Question

I am deriving a Weight update for a simple toy network with a Sigmoid Output Layer. I need some help double checking my math to make sure I did it correctly. I am using Cross-Entropy Loss as my Loss ...
1
vote
1answer
47 views

Why doesn't the optimizer just look for stationary points of the loss function?

I want to have a better understanding of the weight-optimization process. I understand the optimizer(e.g., gradient descent) looks for the direction in which to move the parameters to minimize the ...
1
vote
0answers
21 views

Science practice: Where to introduce approximations?

In my work, I am using an algorithm which relies on estimates of the gradient of the log-posterior at a collection of Monte Carlo samples. Since this gradient is not available in closed form, I must ...
0
votes
1answer
13 views

Should gradient descent based algorithms return the best score?

While studying gradient descent based optimizers I have been wondering if they should return the value after the last update or the value that provides the smallest loss. Pros: returning the best ...
1
vote
1answer
73 views

vanishing gradient and gradient zero

There is a well known problem vanishing gradient in BackPropagation training of ...
0
votes
0answers
22 views

why are squares of gradient used in adaptive learning rates techniques like rmsprop? [duplicate]

I have already had a look at this question Please elaborate the answer and would further like to add, what if we don't use squared derivatives.
0
votes
0answers
29 views

Score function of multi-variate normal distribution

There is an existing question about the score function of multivariate-normal distributions (i.e. the gradient of the log-pdf), but it is somewhat specific to the problem of the OP, and I am ...
0
votes
0answers
21 views

Conceptual explanation of taking derivative of sigmoid function in neural network

In a neural network, we have a bunch of inputs and corresponding weights + a bias which are represented by a multivariable equation. Now we squash this whole equation with a sigmoid function. How ...
0
votes
0answers
16 views

Deriving vectorized form of linear regression

We first have the weights of a D dimensional vector $w$ and a D dimensional predictor vector $x$, which are all indexed by $j$. There are $N$ observations, all D dimensional. $t$ is our targets, i.e, ...
2
votes
1answer
32 views

How Gradient Descent is used for classification with Decision Trees?

I'm not able to see how do we use gradient descent to minimize the loss of binary classification with decision tree. What I understood is that we first have a model (decision tree) that try to predict ...
2
votes
1answer
52 views

Understanding mini-batch gradient descent

I would like to understand the steps of mini-batch gradient descent for training a neural network. My train data $(X,y)$ has dimension $(k \times n)$ and $(1 \times n)$, where $k$ is the number of the ...
1
vote
0answers
74 views

Bias in Gradient Descent (GD) and Stochastic GD (SGD)

Let $\theta$ be weight parameters and assume the loss function to be $L_N(\theta)=\frac{1}{N}\sum_i f(\theta; x_i,y_i)$. Assume a mini batch loss function with a batch of size $M$ and denote the loss ...
0
votes
0answers
11 views

Non - convergence of loss function in Neural Networks with a step activation function [duplicate]

In the below lecture by Professor Patrick H. Winston on Neural Networks: https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-034-artificial-intelligence-fall-2010/lecture-videos/...
2
votes
0answers
51 views

Gradient Descent diverges when input data changes

I'm trying to use the gradient descent to solve the following min problem: $$\min_R \sum_{j=m}^K|| X_j - RX_{j-i}||_2^2, X\in R^{n \times K}$$ For simplicity, assume $m=i+1,\text{ and } i = 1$. That ...
4
votes
3answers
232 views

Basic preconditioned gradient descent example

I'm exploring preconditioned gradient descent using a similar toy problem described in the first part of Lecture 8: Accelerating SGD with preconditioning and adaptive learning rates. I have the ...
0
votes
0answers
13 views

How do you find the most optimal learning rate for gradient descent?

When using gradient descent to solve optimization problems, how do you determine the optimal learning rate? From what I have read in the past, the procedure seems to be "guess and check" ...
1
vote
1answer
86 views

Hessian optimization (Newton method) using the direction given by the gradient to make the next iteration step of the parameters

Reading Deep Learning Book (page 86) I am having trouble understanding the reasons behind using the gradient ($g$) as the direction of the step of the parameters ($x$). I understand that the Newton ...
2
votes
0answers
21 views

Is gradient descent guaranteed to converge to a local minimum (if it doesn't diverge)?

A few times in the literature, I have seen it suggested that higher learning rates can be bad because gradient descent may approach the neighbourhood of a minimum, but then "bounce around" ...
1
vote
0answers
25 views

the loss suddenly was stuck to a very large value after having several epoch

When I changed the final activation layer in the same model from softmax to sigmoid in order to multilabel classification, the loss would suddenly get stuck at a very big value after several epoch. I ...
0
votes
1answer
25 views

In Gradient Boosting over Decision Trees what does it mean to reconstruct the residual

So I am trying to understand what happens in GBDT and particularly I want to know what it mean to "reconstruct the residual." The way I understand it is that the next tree will use the ...
2
votes
1answer
45 views

Linear function gradient descent

I am trying to implement a gradient descent algorithm for a simple linear function: y(x) = x Where initial hypothesis function is: ...
0
votes
1answer
14 views

Initializing gradient boosting with the sample mean

For gradient boosting in the regression setting, the final vector of fitted values is $$F_M(x) = \bar{y} + \rho_1h_1(x) + \ldots, + \rho_M h_M(x)$$ Suppose I have a new data set $x_{new}$ that I'd ...
1
vote
1answer
17 views

Why there is theta in index of gradient symbol in gradient descent update formula for MAML?

In this MAML paper, they use following formula of gradient descent update (see page 3, algorithm 1): $$ \varTheta '\ =\varTheta \ −\ \alpha \nabla _{\varTheta }\mathcal{L}_{\mathcal{T}_{i}}( f_{\...
1
vote
0answers
17 views

Relation between test and train error with gradient descent iterates

My question is about establishing an inequality between population error and expected training error (i.e, expected training error < population error) for a model trained with gradient descent on a ...
1
vote
0answers
23 views

Stopped by zero step from line search - R stops optimization early

I am trying to minimize an objective function, $J(\theta)$, with respect to $\theta$, a 19-dimensional parameter vector. $J(\theta)$ is a smooth nonlinear function so I have tried various gradient-...
0
votes
0answers
19 views

Gradient Descent for changing regression coefficients to minimize error

I recently saw someone input their existing linear regression model into a script that used Adagrad to "tweak" the model's coefficients in order to slightly minimize the model's error. I've ...
2
votes
0answers
25 views

How does scaled conjugate gradient work in neural network training? Comparison with gradient descent

I am very new and beginner in the machine learning world, and I would like to ask if someone could simply explain to me how does the scaled conjugate gradient method work in neural network training? ...
1
vote
0answers
28 views

Why are we interested in gradient with respect to input?

I am learning about sampling methods for Deep Embedding Learning. I was reading an article named: "Sampling Matters in Deep Embedding Learning" (https://arxiv.org/abs/1706.07567). In the ...
3
votes
0answers
33 views

Are there algorithm for finding a minimum of a separately convex function (i.e., $f(x,y)$ convex in $x$ and $y$ but not in $(x,y)$)

Suppose that $f: \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$ is separatly convex function. That is, for a fixed $y$ the mapping $x \to f(x,y)$ is convex and for a fixed $x$ the mapping $y \to f(...
2
votes
0answers
20 views

What are the state of the art optimization methods for neural networks?

Neural networks are usually trained with first order gradient methods and it's variations such as: batch gradient descent, stochastic gradient descent, momentum based gradient and so on.. However ...
0
votes
0answers
48 views

Why not use line search in conjunction with stochastic gradient descent?

I'm familiar with numerical optimization in Engineering context. I have taken several graduate level engineering optimization and operations research courses. I'm beginning to learn machine learning. ...
0
votes
0answers
27 views

Various Methods to Calculate Linear Regression [duplicate]

I have just started learning Machine Learning and one of the very first topics that I have encountered in this venture is Simple Linear Regression. From Andrew Ng's course, I have learned to perform ...
1
vote
1answer
46 views

Mean Square Error and Gradient Descent

I am trying to learn gradient descent and in the course of so I am trying to find the optimal m and c value for my model, for $y=mx+c$ For that, I have plotted the MSE using the below code in python <...
1
vote
1answer
14 views

What's the meaning of these notations in the cost fuction?

Except for the summation, I'm having a hard time figuring out the meaning of these notations. As I assume this is generic and the context is not that necessary, I'm here asking for help. (also, is ...
2
votes
2answers
58 views

Understanding cross entropy loss [closed]

The formula for cross entropy loss is this: $$-\sum_iy_i \ln\left(\hat{y}_i\right).$$ My question is, what is the minimum and maximum value for cross entropy loss, given that there is a negative sign ...
0
votes
1answer
31 views

Why divide the sample size in minibatch gradient descent

Take linear regression for instance the loss is usually define as $L = \frac{1}{2N}\sum_{i = 1}^{N}(Y_i - WX_i)^2$. Two here, as I understand, is to make the derivative look nice. But I am not sure ...
1
vote
1answer
13 views

Does gradient descent work for tabular Q learning?

Suppose I have a tabular Q learning problem such as grid-world. Let our loss be defined as, $$\hat{L}(Q)=0.5(Q(s,a)-(r+\gamma\max_{a'}{Q(s',a')}))^2$$ Then $Q_{k+1}(s,a) = Q_k(s,a) - \eta \nabla \hat {...
2
votes
0answers
36 views

Mini-Batch Gradient Descent - Why does sampling with replacement work?

When sampling the data, either one at a time (as in online learning), or in mini-batches, there exist gradient descent methods which sample with replacement and without replacement. For Mini-Batch ...
1
vote
0answers
71 views

What is wrong with my approach on a custom way of creating Gabor-filter convolution kernels?

Disclosure: I am not a prominent mathematician (current bachelor student) like others on this website and my approach has been mostly pragmatic. Please do tell me if I can improve the formulation of ...
1
vote
0answers
13 views

Momentum vs adaptive step methods

My understanding is that: With momentum, one can avoid e.g. "zig-zags" during gradient decent by averaging gradients to determine a better direction of descent. With adaptive step size ...
0
votes
0answers
11 views

Proximal gradient descent for “projective” $l1$ term

Proximal gradient descent is employed when you want to find $\min f$ where $f(v)=g(v)+h(v)$, $g$ is smooth and $h$ is not smooth. When $h$ is a lasso term of the form $h(v)=\lambda ||v||_1$, the ...

1
2 3 4 5
16