Questions tagged [gradient-descent]
Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.
961
questions
2
votes
1
answer
18
views
Does normalizing/changing the scale of the target variable impact the shape of the loss function equation?
I was under the impression that changing the scale/normalizing the target variable in a regression task would not change the overall shape of the loss function equation but would simply translate/move ...
1
vote
0
answers
19
views
Could someone help me interpret the data that I have gathered thus far?
I am trying to train a SVM model for my statistical learning course. The problem is a binary image classification problem (wildfire, nowildfire). This is the rigorous amount of testing that I have ...
- 111
0
votes
0
answers
9
views
Gradient of multiclass hinge loss (max of max difference version)
I want to train a linear classifier for image classification. I have a $\mathbf{W}$ of shape $D\times K$ where $D$ is the dimension of the vectorized version of the image (including bias, herein 3073) ...
- 220
0
votes
0
answers
53
views
How did deep learning overcome numerical problems associated with earlier ANNs? [closed]
From my understanding, the basic design of an artificial neuron has remained essentially the same since the 1960s. Before the bloom of deep learning models in ~2010, there were two obstacles to deep ...
- 101
0
votes
0
answers
30
views
Target binning in regression
I would like to find a predictive density for target variable via multi-class classification. Suppose we are given a set of features $\mathbf X$ and continuous target $\mathbf y$. Replace each $y$ ...
- 509
1
vote
0
answers
38
views
Neural network parameters dependency vs gradient descent [duplicate]
Neural network parameters are not independent from each other. How do we account for this dependence in the gradient descent algorithm? Intuitively I would expect that if we first change the weights ...
- 371
0
votes
1
answer
24
views
Why do the error derivatives become small if we start with a large learning rate?
In these slides from Hinton (https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf) there is this statement:
I don't understand why "The error derivatives for the hidden units ...
- 1
0
votes
0
answers
30
views
My loss has a non-differentiable point
I had to design a loss function max(0,x). It's not differentiable at x=0. In order to train it with gradient descent, what should I do?
I have learned that subgradient can be used instead, so does it ...
1
vote
1
answer
12
views
Batch Normalization derivatives
I'm following the derivative calculation of Batch Norm paper:
Something doesn't seem right. In the 3rd equation shouldn't we lose the 2nd term as the sum is equal to 0 ($\mu_B$ is the mean of the $...
- 2,869
0
votes
0
answers
7
views
Are some gradient weights equal?
I want to create a 3 layers neural network from scratch to perform linear regression.
The first and the second layer have 2 neurons, and the last layer has one neuron.
Feature vector x is divided into ...
- 1
0
votes
0
answers
10
views
How can you detect if your Gradient Descent algorithm is in a local minimum and not the global minimum?
For both batch and stochastic gradient descent, how can you detect if it is stuck in a local minimum and not the global minimum?
Is there a sophisticated way to do this, as opposed to guess and check ...
- 523
0
votes
0
answers
12
views
Calculating initial values of a Linear Regression Model
I have the following problem
Given the training data for a linear regresison problem as follow:
Input
Output
0
0
1
2
-1
-2
2
3
After the first iteration, the values of the two coefficients are ...
1
vote
0
answers
19
views
Should I destandardize the errors from training the neural network?
So, I am learning a bit about Neural Networks. I have built a code in PyTorch for regression, and I have standardized both the features and the target variables following this answer. My question is ...
1
vote
0
answers
10
views
AdaGrad: motivation and complex values
I have started learning the math behind the AdaGrad optimizer, and two questions emerged that I cannot find any answers on the Internet:
It is said that in AdaGrad "parameters associated with ...
1
vote
0
answers
14
views
Power Regression Unit Conversion
Say I have fit parameters $a, b, c$ for a power regression model (using SGD and a MSE loss),
$$y = ax^b + c.$$
Suppose the derived $a, b, c$ fit the data very well. Now suppose $x$ undergoes a change ...
- 111
0
votes
0
answers
16
views
Finding the gradient for logistic regression using sign outputs
In logistic regression, if we use the sign outputs, such that $y \in \{-1,1\}$, we have that the loss function is given by (from [here])
$$L(y,\beta^Tx)=\log(1+\exp{(-y\cdot \beta^Tx}))$$
In this case,...
- 95
0
votes
0
answers
24
views
Why is the average used when combining the gradients of batch or mini-batch gradient descent?
When you do batch gradient descent in neural networks, you find the gradients for all the inputs in a batch and average them out. Why does the arithmetic mean work in finding the optimal gradient? Why ...
- 215
0
votes
0
answers
28
views
Why unbounded above activation function is important for training
One of the desirable properties of activation functions is to be unbounded above and bounded below. I guess part of the reasons why it should be unbounded above is to avoid vanishing gradient problems ...
- 249
2
votes
1
answer
160
views
Can I use Log Loss to solve a non-linear Classification task?
Assume the following data, where the yellow points represent class $1$ and the purple points represent class $2$. I would like to know, if it is possible to build a linear classifier (sigmoid) by ...
- 351
1
vote
0
answers
28
views
Does the transformation of variables affect the performance of gradient descent?
Suppose we want to minimize the objective function $f(x)$ with respect to $x\in \mathbb{R}^p$. Wtih a linear transformation of $x$, e.g., $z = Qx$ for a proper $Q\in\mathbb{R}^{p\times p}$, the new ...
- 35
0
votes
0
answers
13
views
Can SGD be applied in this setting?
I am trying to built some simple setting to understand better stochastic gradient ascent/descent and I got stuck with the following reasonings.
Suppose we have two r.v. $X,A$ with (unknown) joint p.d....
- 808
0
votes
0
answers
68
views
What is the meaning of uncentered variance? [duplicate]
I read some articles that describe ADAM optimizer and they used uncentered variance expression.
I'm not sure I understand the ...
- 125
1
vote
1
answer
56
views
What does the ellipse in Gradient descent describes?
I saw multiple articles describes GD or SGD with the following diagram:
I didn't saw any explanation about the ellipses.
What ...
- 125
1
vote
0
answers
40
views
Why is Gradient Accumulation not used frequently in training large models compared to using bigger batch sizes?
So I was currently going through various implementations of models and they set a large batch size of around 256 , running this is google collab is very memory intensive, so I decided to use a batch ...
3
votes
0
answers
77
views
Is there any advantage from using Momentum Schedulers in training models using SGD than using a constant momentum of 0.9?
Recently I noticed that some pytorch repos of papers use Learning Rate Scheduler and momentum rate Scheduler , a lot of momentum rate schedulers exist similar to LR scheduler ranging from Lambda, ...
2
votes
1
answer
206
views
Is this "stochastic gradient method" different from the stochastic gradient descent algorithm?
In a computational statistics book, I found an optimization method to find local minimum of a function.
Let's assume that we have a differentiable function $f: \mathbb{R}^2 \longrightarrow \mathbb{R}$....
- 135
1
vote
1
answer
30
views
Gradient descent and epoch
Suppose our hypothesis space is $$\mathcal{H}=\{f:f(x)=f_\theta (x), \theta\in \Theta\},$$ where $\theta$ is the trainable parameter.
Suppose we have a dataset $\{x_i,y_i\}_{i=1}^N.$
In the notes from ...
- 177
0
votes
0
answers
21
views
how does norm calculation helps in gradient descent?
I have studied few equations precisely the equations catering multi-objective gradient descent that calculates the norm of the jacobian matrix and minimizes the same. I would like to know the ...
- 109
0
votes
0
answers
16
views
How does changing GAN training method afftects the GAN?
I am training a GAN in which I am optimizing or reducing the loss of the generator and the discriminator simultaneously. However, the images generated are very noisy. What could be the hidden reason? ...
- 109
1
vote
1
answer
55
views
Why do we need stochastic gradient descent when we can precompute certain results in batch gradient descent?
I have read that batch gradient descent forces this summation at every step of the update, which makes it time consuming.
But if we have the following hypothesis function: $$h(x^i) = w_0 + w_1x^i$$
...
- 13
0
votes
0
answers
29
views
How is the saddle point in a multi-objective optimization different from a saddle point in a single objective optimization?
I am doing gradient descent with the multiple objectives. For example I am optimizing two functions f1 and f2 simultaneously. The multi-objective optimization is done to get the pareto optimal points. ...
- 109
0
votes
0
answers
10
views
Gradient descent for spectra unmixing not converging
I'm trying to a decompose the spectrum of a multi-fluorophore system into the weighted known component spectra.
Basically, the multi-fluorophore spectrum is a weighted linear combination of the ...
- 1
2
votes
0
answers
259
views
Which methods can improve model fit through gradient descent?
I am looking to solve the following question:
Which methods below can improve model fit (MSE) through gradient descent?
Use a high learning rate
Use stochastic gradient descent
Start with a high ...
- 21
1
vote
1
answer
61
views
Can the exact same split occur in subsequent trees in a gradient boosted trees model?
I am aware that there are several questions about feature/split selection in gradient-boosted trees (e.g. If a feature has already split, will it hardly be selected to split again in the subsequent ...
- 13
1
vote
0
answers
38
views
Please help me to understand the Taylor’s theorem when transiting from Gradient Boost to XGboost
I am reading this article, which explains how the algorithm replaces the actual loss function with so-called 2nd order Taylor expansion. I can understand til Step 4, and can't understand step 5.
I ...
- 111
5
votes
2
answers
617
views
Gradient Descent and Learning Rate
Assuming that our function is convex and only has a single minimum. Would gradient descent ever hit the minimum if the learning rate was extremely small? Or would it bounce back and forth without ever ...
- 109
0
votes
0
answers
13
views
How training GANs with the method of optimizing 2 loss functions simultaneously differ from the actual alternate training?
I am training GAN in a multiobjective optimization setting where I am optimizing both the loss functions(generator and discriminator) at the same like optimizing 2 functions simultaneously. However, ...
- 109
1
vote
0
answers
56
views
Using Gradient Decent For PCA Optimization [closed]
I'm trying to solve the PCA problem:
For $k\in N$ some number and $X\in R(n\times d)$ where I'm trying to find $w\in R(k\times d)$ such that:
$w = argmax( E(WXX^T))$
(I might be wrong with the ...
- 11
1
vote
0
answers
40
views
Gradient Descent Algorithm: For multiple local minimum which one to pick
This might be a newbie question, but it is from a newbie.
If there are multiple local minimums, and the function converges at various local minima, which local minima to pick for optimization? Do we ...
- 111
1
vote
1
answer
61
views
gradient ascent vs gradient descent update rule
I'm trying to understand the differences between the update rule for stochastic gradient ascent and descent. I've read some articles and still don't understand how to calculate the update rule:
...
- 11
1
vote
1
answer
305
views
Why does SKLearn's Logistic Regression model have the same coefficients as my own model for 1 class but have different coefficients for other classes
I am currently implementing logistic regression from scratch and I'm comparing my model with SKLearn's logistic regression. Since this is just an exercise, I decided to use toy data, specifically ...
- 11
1
vote
1
answer
163
views
How error derivative becomes zero in gradient descent
Previous questions this & this does not answer my question
Code
...
- 667
1
vote
1
answer
22
views
why we use same learning rate in the whole process of the gradient descent?
In theory, we know while we are descending to the point where the error is zero, we give big steps that are learning rate will be big. And when we are near to the error equal to zero we start giving ...
- 667
0
votes
0
answers
18
views
Is gradient descent for non-parametric maximum likelihood estimation? [duplicate]
In my reading of maximum likelihood estimation, they go through samples with KNOWN distributions (e.g. binomial, poisson, etc.). I wonder how can I connect to my knowledge of machine learning.
In ...
- 355
1
vote
0
answers
28
views
Gradient Descent for Multi-Level / Mixed / Hierarchical Regression Model
How would gradient descent work in a multilevel regression setting? This is fairly clear to me in a standard linear regression formulation, but haven't been able to wrap my head around parameter ...
- 11
1
vote
1
answer
183
views
Subgradient for sparse-group lasso
Sparse-group lasso is defined as
$$\frac{1}{2n}\left\|y-X\beta \right\| + (1-\alpha)\lambda\sum_{l=1}^m \sqrt{p_l}\left\|\beta^{(l)} \right\|_2 + \alpha \lambda \left\| \beta\right\|_1$$
In the SGL ...
- 95
6
votes
2
answers
507
views
ALS vs SGD in parallelization
So given the standard objective in matrix factorization for collaborative filtering of minimizing:
$$
L = \sum_{u,i \in S} (r_{ui}-q_i^Tp_u)^2 + \lambda(\sum_i||q_i^2||+\sum_u||p_u^2||)
$$
, where $r_{...
- 311
2
votes
1
answer
132
views
Why do we regularize large gradients corresponding to large errors?
While reviewing some scientific blogs, I found them recommending using gradient clipping for large error gradients. However, intuitively one would think that when model predictions are completely off, ...
- 448
1
vote
2
answers
122
views
How does epoch-wise double descent occur if training error is 0?
In this paper, they talk about the existence of epoch-wise double descent. In Figure 10, you can see that, with a sufficiently large model, the test error keeps decreasing even after the training ...
- 13
0
votes
0
answers
72
views
Gradient descent / Adam converging to suboptimal solutions
I am using neural nets to find the minimum of a complex function to which I compute the mean (crit in my code). Here is my net :
...
- 1