Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Deriving the gradient in Batch Normalization with respect to weights

At the bottom of page 2 of the paper “L2 Regularization versus Batch and Weight Normalization”, the equation for the gradient of the output with respect to the weights is given as: $$ \triangledown ...
0
votes
0answers
15 views

For linear regression, do different hypothesis equations change the gradient descent equation I need to use?

For example, take the hypothesis function $h(x) = \theta_0 + \theta_1x_1 + \theta_2x_2$ When plugged into the MSE equation I get $J(\theta_0, \theta_1, \theta_2) = \frac{1}{2m} \sum_{i=1}^{m}(\...
0
votes
0answers
19 views

Vanishing gradient appears from the first iteration

I made a neural network for regression problem. It has 9 convolution layers and two fully connected layers at the end. All layers have ReLU activation function except the last FC layer. This is the ...
1
vote
1answer
34 views

How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
1
vote
0answers
17 views

How to minimize the sum of Frobenius norm and Nuclear norm

I have to minimize an objective function of the the form : $||X_{s} - Y_{s}D_{s}||_{F}^{2} + ||D_{s}||_{F}^{2} + ||D_{s}||_{*}^{2}$ where $||.||_{F}$ denotes the Frobenius norm and $||.||_{*}$ ...
0
votes
0answers
4 views

General method to add feature selection to an objective function?

In general, if you have an objective function that is differentiable with regards to its parameters you can apply gradient descent to minimize it (and even if it's not using a variety of possible ...
1
vote
0answers
19 views

Hessian Matrix for MultiClass Softmax in Gradient Boosting (XGBoost): $2p_i(1-p_i)$

In the context of MultiClass Softmax, for a particular training instance, label and prediction $y, p \in \mathbb{R}^K$ (K categories). The hessian matrix for Multiclass SoftMax with K categories is a $...
0
votes
1answer
28 views

cross-validation with batch gradient descent

I have a question about using batch gradient descent along with cross-validation. When using batch gradient Descent, data is split into batches which are used to train the model and update the ...
0
votes
0answers
25 views

How to compute gradient of neural net

I can't come up with with gradient. I've been trying for a couple hours now but I can't get it right. I'm building a neural net with the following properties: two class classification, 1 hidden layer ...
2
votes
1answer
44 views

Decomposing Gradient Decent Error in Eigenvector Space

I'm going through Why Momentum Really Works and am unable to understand the following line in the article. "By writing the contributions of each eigenspace’s error to the loss $$f(w^{k})-f(w^{\star})=...
0
votes
0answers
23 views

Reinforcement Learning - Value Function Approximation

I am new to Reinforcement (Machine) Learning; I started my project with Q Learning (Tabular Q), which was easy to understand. I am now trying to write the code for Value Function Approximation. I ...
1
vote
2answers
40 views

is there a “generic” gradient descent

on this week we learned that the general form of the update step for gradient descent is: $x := x - \alpha \frac{\partial f}{\partial x}$ So, in order to find x where f is minimum, we have to ...
8
votes
3answers
154 views

How to test for the best parameters for transformed independent variable in linear model

Let's assume that I have a linear model with $k$ variables: $y = \beta_0 + \beta_1\cdot x_1 + \dots + \beta_k \cdot x_k$. Now, I want to add variable $x_{k+1}$, but, according to domain knowledge, ...
3
votes
1answer
35 views

Distribution of gradients across dimensions for neural networks

Does the distribution of gradients for neural networks known to follow a particular distribution? That is, suppose I've a model with $N$ parameters. Then, the (stochastic) gradient at some point is a ...
2
votes
0answers
20 views

Hill Climbing with hysteresis

I'm trying to solve a specific problem related to my work in experimental physics. However, I'll try to keep my question as general as possible so that it is useful to a wider audience. If some ...
0
votes
1answer
39 views

Why using gradient decent if we can just minimize function in closed form?

I don't really understand why gradient descent is so important in neural networks? Wouldn't it be much easier to define an objective function in a way to do ...
0
votes
1answer
46 views

using finite difference to estimate high dimensional gradient in gradient descent methods

I'm not very familiar with optimization problem, but I know that if the gradient of function is hard to find, it can use finite difference method to estimate it. Like scipy.minimize, it would use this ...
0
votes
0answers
8 views

Gradient of a convex loss for linear classifiers

Let L(w; x; y) be a convex loss function for a linear classifier w. Can you always express the gradient of ...
1
vote
0answers
14 views

Mathematical proof of tradeoff between estimation error and computation cost in mini-batch gradient descent

" with more examples, the estimate would have a lower standard error, but the return is less than linear compared to the computational burden we incur." Came across this line while studying one ...
0
votes
1answer
29 views

Interpreting cost change plot in a neural network for learning XOR

I tried to build a neural net for learning XOR. The design is as follows: 1st layer: compute linear function of input 4:2 with 2:2 weights and adding 1:2 bias. 2nd layer: apply sigmoid to all ...
0
votes
0answers
33 views

Why is the step length by default equal to 1 in gradient boosting?

On ESL p.359, it explains steepest descent: But in 10.37, it is trying to minimize the distance to g_im. It looks like the default step length is 1. Why is it so?
0
votes
1answer
22 views

Do we have guarantees about Adam's convergence when we reach an region with gradient $0$?

Recall the Adam update rule: $$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$ $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ $$\hat{m}_t = \dfrac{m_t}{1 - \beta^t_1}$$ $$\hat{v}_t = \dfrac{v_t}{1 - \...
0
votes
1answer
30 views

Decision tree that fits a regression at leaf nodes

Is there any academic work on any Decision Tree that fits a regression at its final leaf node? For instance, suppose I have 100 features (X), and use them build a tree with 3 depths such that I have ...
1
vote
0answers
22 views

Logistic regression fitting methods clarification

Each book I read propose a different fitting method for Logistic Regression. The general idea is to maximize this expression. $$ Pr\left(\beta|y,X,M\right) = \frac{Pr\left(y|\beta,X,M\right) Pr(\...
0
votes
2answers
31 views

Neural network training: skipping layers when performing weight updates

Suppose you had many models that are trained to classify a sequence of logical characters into 'satisfiable' or 'unsatisfiable' (e.g., the sequence "A and notA" is unsatisfiable). Suppose all of these ...
1
vote
1answer
73 views

Why is local-m Attention differentiable?

In this paper (https://arxiv.org/pdf/1508.04025.pdf) local attention is introduced. With local-m attention, we compute the attention vector over the source hidden states within a window around the ...
4
votes
1answer
289 views

Gradient descent: Shouldn't step size be proportional to inverse of gradient of residual?

It has been decades since I coded up any type of gradient descent algorithm to drive a function to zero (or to a minimum). I am following this tutorial, which minimizes $J(\overrightarrow{\theta})$. ...
0
votes
1answer
46 views

Why does gradient descent fail training a network for predicting times table?

I am training a feedforwardnet with gradient descent traingd as backpropagation algorithm to predict times table. ...
3
votes
0answers
48 views

Why does gradient descent outperform closed form solution

In Andrew Ng's machine learning course, in exercise 3 he uses gradient descent (or some other iterative algorithm?) to find coefficients for a problem of handwritten number classification. I repeated ...
0
votes
0answers
21 views

Backpropagation for Linear Softmax classifier

I'm currently implementing a Linear Softmax classifier from scratch where $\mathbf{\hat y} =\mathbf{x^TW}$. I'm not sure about the backpropagation step, however. $L$ denotes the Cross Entropy loss ...
1
vote
1answer
35 views

Neural Network Batch Training

I have read a few threads on implementing batch training in neural networks. Still I don't understand some specifics of the implementation: When backpropagating the accumulated error of a given batch ...
2
votes
0answers
27 views

What are typical magnitudes of gradients in neural networks?

We've all heard about the vanishing and exploding gradient problems. But I'm having a hard time finding any concrete numbers like "this gradient is x which is way ...
0
votes
0answers
124 views

Deriving Gradient from negative log-likelihood function

I have been having some difficulty deriving a gradient of an equation. I have a Negative log likelihood function, from which i have to derive its gradient function. Negative log likelihood function ...
1
vote
0answers
48 views

Given joint density function, derive $\alpha$ and calculate maximum likelihood formula

Given the join density function $$ f(X_1 = x_1, X_2 = x_2, X_3 = x_3) = \alpha\cdot exp[{\eta _1 x_1+ \eta _2x_2 + \eta_3x_3 - w_{12}x_1x_2- w_{13}x_1x_3- w_{23}x_2x_3]} $$, where the parameter set $\...
0
votes
2answers
67 views

Why do people say gradient descent is slower than stochastic gradient descent? That's obviously not true?

With gradient descent, you calculate the gradient for the entire sample at once. With SGD, you calculate it on each sample, and then you do the same for every other sample, until you have done 1 full ...
1
vote
1answer
19 views

does cost function of one hidden layer perception have only one global minimal

is it true that: After training a multilayer perceptron with one hidden layer using gradient descent, we always get the same decision boundary regardless of the initialization point.
1
vote
1answer
32 views

alternating negative and positive value of slope and y-intercept in gradient descent

I'm working with the following code for gradient descent for simple linear regression: ...
0
votes
1answer
376 views

How to plot cost function against iterations? [closed]

I am new to coding in machine learning. I am trying to plot a graph for the gradient descent of a univariate function. ...
0
votes
0answers
59 views

How does Feature Scaling help Gradient Descent? [duplicate]

I am following deep learning.ai's videos on Coursera. I have a couple of questions about feature scaling using the formula: $$ (x - \mu)/ \sigma $$ Edit: There are similar questions which deal with ...
0
votes
1answer
129 views

Help understanding Vanishing and Exploding Gradients

I am following deeplearning.ai's videos on Coursera. I have a couple of questions regarding vanishing and exploding gradients. The following is Prof Andrew Ngs lecture slides: From what Prof Ng says ...
0
votes
0answers
22 views

Why is the stochastic gradient of a layer almost orthogonal to its weight?

In the paper Fixup Initialization: Residual Learning Without Normalization. In Page 5 when talking about effects of multipliers, the authors mentioned that Specifically, as the stochastic gradient ...
0
votes
0answers
13 views

show convergence rate of gradient decent of a quadratic form is bounded by some function

Geven a quadratic form $f(x)=\frac{1}{2}x^TAx+x^Tb+a$, where $A$ is a symmetric positive definite matrix. We use gradient decent to compute the global min by $x_n=x_{n-1}-\triangledown f(x_{n-1})$ and ...
3
votes
2answers
100 views

Why is the expected gradient of a density not parallel to the expected gradient of the log density?

I'm confused by a seemingly counter-intuitive property of the interaction between distributions, log transforms, expectations and gradients. Suppose I have some distribution over random variable $x$ ...
2
votes
0answers
63 views

Gradient Descent: Taking the derivative of the raw error? [closed]

I'm reading this book called "Grokking Deep Learning" and there's this one part I can't help but be skeptical of: Here, the graph plots error squared against weight. The book however calculates the ...
0
votes
0answers
18 views

Gradient on subset of training data is proportional to the true gradient?

I have been thinking of proving the following: Prove that the gradient calculated on a random subset of a training set on average is proportional to the true gradient. However, proving is not my ...
2
votes
0answers
78 views

Proving Linear Regression with Gradient Descent Converge to OLS estimates

Problem I am having trouble showing that the parameters $\theta\in \mathbf{R}^{m}$ for Linear Regression converge to the classic OLS estimates using gradient descent. Please find below my attempt: ...
0
votes
0answers
443 views

Linear Regression in Python using gradient descent

(cross-posted from data-science StackExchange)(someone recommended that this community is more appropriate for my problem) I am trying to implement a simple multivariate linear regression model ...
3
votes
1answer
347 views

How can one implement PCA using gradient descent?

I have to implement PCA using gradient descent and stop at convergence. I am not able to find the objective function. I know that the aim of PCA is to reduce the $n$-dimensional matrix to $k$ ...
0
votes
1answer
613 views

Gradient descent with Binary Cross-Entropy for single layer perceptron

I'm implementing a Single Layer Perceptron for binary classification in python. I'm using binary Cross-Entropy loss function and gradient descent. The gradient descent is not converging, may be I'm ...
0
votes
1answer
47 views

Finding best fit curve with unknown power

If I was to build a program to estimate the best curve fit of type $a * x^b$ where a and b are the parameters I'm optimizing, what would be my go to methods? I know I can use ordinary least squares ...

1 2 3 4 5 15