Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
0
votes
2answers
35 views

Why do earlier hidden layers learn slower?

I'm reading chapter 5 of Nielsen's textbook about vanishing gradients. He states: In at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden ...
0
votes
0answers
4 views

gradient computation in Nesterov momentum

I am reading deeplearningbook by Ian Goodfellow et al. And I have a question about gradient computation in Nesterov momentum. There are two pages from this book, which describe Nesterov momentum: ...
0
votes
0answers
15 views

Investigator efficiency computation

I need a little help with inferring ground truth and investigation capabilities from data. I have an incomplete matrix consisting of binary decisions taken by investigators on documents i.e. for every ...
0
votes
0answers
24 views

Valid use of differentiable almost everywhere functions, like hinge loss, in gradient optimization/learners, like SciKit-Learn's SGDClassifier?

So, my abstract question is: is it valid (in the sense that stable convergence is roughly expected) to use functions that are differentiable almost everywhere in the practical application of ...
0
votes
0answers
12 views

Reference request for normalized gradient descent

Can someone introduce a good article/textbook explaining variants of the gradient descent method? In particular, I am interested in the normalized gradient descent where one works with $\frac{\nabla f}...
1
vote
0answers
31 views

Can gradient descent work iteratively instead of simultaneously?

When multivariate gradient descent is updating weight of single feature, why doesn't it project this new weight when updating weights of next features? Let's say we have this example of gradient ...
0
votes
1answer
21 views

How to derive the gradient of the reparameterized score function estimator?

In the paper Evolution Strategies as a Scalable Alternative to Reinforcement Learning, the authors derive the following gradient of the score function estimator $$ \begin{align} \nabla_\psi\mathbb E_{...
0
votes
0answers
17 views

FGSM vs Gradient Descent

FGSM as described by Goodfellow et al takes the sign of the gradient of a model w.r.t. the input. And then performs gradient descent (or ascent, depending on your objective). However why not use ...
0
votes
0answers
25 views

Highly likely reasons for ineffectiveness of logistic regression (accuracy is always 0.5)

I am doing cats-vs-dogs competition. So it's binary classification. I randomly pick pictures (i.g. N=7000), resize them to 100x100 and convert to greyscale. Then I use SGDClissifier with loss='log' (...
0
votes
0answers
7 views

Deriving the gradient in Batch Normalization with respect to weights

At the bottom of page 2 of the paper “L2 Regularization versus Batch and Weight Normalization”, the equation for the gradient of the output with respect to the weights is given as: $$ \triangledown ...
0
votes
0answers
18 views

For linear regression, do different hypothesis equations change the gradient descent equation I need to use?

For example, take the hypothesis function $h(x) = \theta_0 + \theta_1x_1 + \theta_2x_2$ When plugged into the MSE equation I get $J(\theta_0, \theta_1, \theta_2) = \frac{1}{2m} \sum_{i=1}^{m}(\...
1
vote
1answer
44 views

How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
1
vote
0answers
18 views

How to minimize the sum of Frobenius norm and Nuclear norm

I have to minimize an objective function of the the form : $||X_{s} - Y_{s}D_{s}||_{F}^{2} + ||D_{s}||_{F}^{2} + ||D_{s}||_{*}^{2}$ where $||.||_{F}$ denotes the Frobenius norm and $||.||_{*}$ ...
0
votes
0answers
4 views

General method to add feature selection to an objective function?

In general, if you have an objective function that is differentiable with regards to its parameters you can apply gradient descent to minimize it (and even if it's not using a variety of possible ...
1
vote
0answers
44 views

Hessian Matrix for MultiClass Softmax in Gradient Boosting (XGBoost): $2p_i(1-p_i)$

In the context of MultiClass Softmax, for a particular training instance, label and prediction $y, p \in \mathbb{R}^K$ (K categories). The hessian matrix for Multiclass SoftMax with K categories is a $...
0
votes
1answer
59 views

cross-validation with batch gradient descent

I have a question about using batch gradient descent along with cross-validation. When using batch gradient Descent, data is split into batches which are used to train the model and update the ...
0
votes
0answers
25 views

How to compute gradient of neural net

I can't come up with with gradient. I've been trying for a couple hours now but I can't get it right. I'm building a neural net with the following properties: two class classification, 1 hidden layer ...
2
votes
1answer
47 views

Decomposing Gradient Decent Error in Eigenvector Space

I'm going through Why Momentum Really Works and am unable to understand the following line in the article. "By writing the contributions of each eigenspace’s error to the loss $$f(w^{k})-f(w^{\star})=...
0
votes
0answers
24 views

Reinforcement Learning - Value Function Approximation

I am new to Reinforcement (Machine) Learning; I started my project with Q Learning (Tabular Q), which was easy to understand. I am now trying to write the code for Value Function Approximation. I ...
1
vote
2answers
40 views

is there a “generic” gradient descent

on this week we learned that the general form of the update step for gradient descent is: $x := x - \alpha \frac{\partial f}{\partial x}$ So, in order to find x where f is minimum, we have to ...
8
votes
3answers
157 views

How to test for the best parameters for transformed independent variable in linear model

Let's assume that I have a linear model with $k$ variables: $y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k$. Now, I want to add variable $x_{k+1}$, but, according to domain knowledge, ...
3
votes
1answer
36 views

Distribution of gradients across dimensions for neural networks

Does the distribution of gradients for neural networks known to follow a particular distribution? That is, suppose I've a model with $N$ parameters. Then, the (stochastic) gradient at some point is a ...
2
votes
0answers
20 views

Hill Climbing with hysteresis

I'm trying to solve a specific problem related to my work in experimental physics. However, I'll try to keep my question as general as possible so that it is useful to a wider audience. If some ...
0
votes
1answer
42 views

Why using gradient decent if we can just minimize function in closed form?

I don't really understand why gradient descent is so important in neural networks? Wouldn't it be much easier to define an objective function in a way to do ...
0
votes
1answer
60 views

using finite difference to estimate high dimensional gradient in gradient descent methods

I'm not very familiar with optimization problem, but I know that if the gradient of function is hard to find, it can use finite difference method to estimate it. Like scipy.minimize, it would use this ...
0
votes
0answers
8 views

Gradient of a convex loss for linear classifiers

Let L(w; x; y) be a convex loss function for a linear classifier w. Can you always express the gradient of ...
1
vote
0answers
14 views

Mathematical proof of tradeoff between estimation error and computation cost in mini-batch gradient descent

" with more examples, the estimate would have a lower standard error, but the return is less than linear compared to the computational burden we incur." Came across this line while studying one ...
0
votes
1answer
38 views

Interpreting cost change plot in a neural network for learning XOR

I tried to build a neural net for learning XOR. The design is as follows: 1st layer: compute linear function of input 4:2 with 2:2 weights and adding 1:2 bias. 2nd layer: apply sigmoid to all ...
0
votes
0answers
39 views

Why is the step length by default equal to 1 in gradient boosting?

On ESL p.359, it explains steepest descent: But in 10.37, it is trying to minimize the distance to g_im. It looks like the default step length is 1. Why is it so?
0
votes
1answer
22 views

Do we have guarantees about Adam's convergence when we reach an region with gradient $0$?

Recall the Adam update rule: $$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ $$\hat{m}_t = \dfrac{m_t}{1 - \beta^t_1}$$ $$\hat{v}_t = \dfrac{v_t}{1 - \...
0
votes
1answer
35 views

Decision tree that fits a regression at leaf nodes

Is there any academic work on any Decision Tree that fits a regression at its final leaf node? For instance, suppose I have 100 features (X), and use them build a tree with 3 depths such that I have ...
1
vote
0answers
22 views

Logistic regression fitting methods clarification

Each book I read propose a different fitting method for Logistic Regression. The general idea is to maximize this expression. $$ Pr\left(\beta|y,X,M\right) = \frac{Pr\left(y|\beta,X,M\right) Pr(\...
0
votes
2answers
31 views

Neural network training: skipping layers when performing weight updates

Suppose you had many models that are trained to classify a sequence of logical characters into 'satisfiable' or 'unsatisfiable' (e.g., the sequence "A and notA" is unsatisfiable). Suppose all of these ...
1
vote
1answer
86 views

Why is local-m Attention differentiable?

In this paper (https://arxiv.org/pdf/1508.04025.pdf) local attention is introduced. With local-m attention, we compute the attention vector over the source hidden states within a window around the ...
4
votes
1answer
298 views

Gradient descent: Shouldn't step size be proportional to inverse of gradient of residual?

It has been decades since I coded up any type of gradient descent algorithm to drive a function to zero (or to a minimum). I am following this tutorial, which minimizes $J(\overrightarrow{\theta})$. ...
0
votes
1answer
49 views

Why does gradient descent fail training a network for predicting times table?

I am training a feedforwardnet with gradient descent traingd as backpropagation algorithm to predict times table. ...
3
votes
0answers
51 views

Why does gradient descent outperform closed form solution

In Andrew Ng's machine learning course, in exercise 3 he uses gradient descent (or some other iterative algorithm?) to find coefficients for a problem of handwritten number classification. I repeated ...
0
votes
0answers
26 views

Backpropagation for Linear Softmax classifier

I'm currently implementing a Linear Softmax classifier from scratch where $\mathbf{\hat y} =\mathbf{x^TW}$. I'm not sure about the backpropagation step, however. $L$ denotes the Cross Entropy loss ...
1
vote
1answer
38 views

Neural Network Batch Training

I have read a few threads on implementing batch training in neural networks. Still I don't understand some specifics of the implementation: When backpropagating the accumulated error of a given batch ...
2
votes
0answers
27 views

What are typical magnitudes of gradients in neural networks?

We've all heard about the vanishing and exploding gradient problems. But I'm having a hard time finding any concrete numbers like "this gradient is x which is way ...
0
votes
0answers
154 views

Deriving Gradient from negative log-likelihood function

I have been having some difficulty deriving a gradient of an equation. I have a Negative log likelihood function, from which i have to derive its gradient function. Negative log likelihood function ...
1
vote
0answers
49 views

Given joint density function, derive $\alpha$ and calculate maximum likelihood formula

Given the join density function $$ f(X_1 = x_1, X_2 = x_2, X_3 = x_3) = \alpha\cdot exp[{\eta _1 x_1+ \eta _2x_2 + \eta_3x_3 - w_{12}x_1x_2- w_{13}x_1x_3- w_{23}x_2x_3]} $$, where the parameter set $\...
0
votes
2answers
91 views

Why do people say gradient descent is slower than stochastic gradient descent? That's obviously not true?

With gradient descent, you calculate the gradient for the entire sample at once. With SGD, you calculate it on each sample, and then you do the same for every other sample, until you have done 1 full ...
1
vote
1answer
19 views

does cost function of one hidden layer perception have only one global minimal

is it true that: After training a multilayer perceptron with one hidden layer using gradient descent, we always get the same decision boundary regardless of the initialization point.
1
vote
1answer
39 views

alternating negative and positive value of slope and y-intercept in gradient descent

I'm working with the following code for gradient descent for simple linear regression: ...
0
votes
1answer
559 views

How to plot cost function against iterations? [closed]

I am new to coding in machine learning. I am trying to plot a graph for the gradient descent of a univariate function. ...
0
votes
0answers
62 views

How does Feature Scaling help Gradient Descent? [duplicate]

I am following deep learning.ai's videos on Coursera. I have a couple of questions about feature scaling using the formula: $$ (x - \mu)/ \sigma $$ Edit: There are similar questions which deal with ...
0
votes
1answer
184 views

Help understanding Vanishing and Exploding Gradients

I am following deeplearning.ai's videos on Coursera. I have a couple of questions regarding vanishing and exploding gradients. The following is Prof Andrew Ngs lecture slides: From what Prof Ng says ...
0
votes
0answers
22 views

Why is the stochastic gradient of a layer almost orthogonal to its weight?

In the paper Fixup Initialization: Residual Learning Without Normalization. In Page 5 when talking about effects of multipliers, the authors mentioned that Specifically, as the stochastic gradient ...
0
votes
0answers
13 views

show convergence rate of gradient decent of a quadratic form is bounded by some function

Geven a quadratic form $f(x)=\frac{1}{2}x^TAx+x^Tb+a$, where $A$ is a symmetric positive definite matrix. We use gradient decent to compute the global min by $x_n=x_{n-1}-\triangledown f(x_{n-1})$ and ...

1
2 3 4 5
15