# Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

934 questions
Filter by
Sorted by
Tagged with
7 views

### How is the loss function related to the derivative of a specific output neuron?

Suppose we have an output vector of three values to give a concrete example: [0, 1, 0.8]. Suppose the ground truth values are [1,1,1]. MSE loss will return about 0.35. How does the value of 0.35 ...
7 views

### How training GANs with the method of optimizing 2 loss functions simultaneously differ from the actual alternate training?

I am training GAN in a multiobjective optimization setting where I am optimizing both the loss functions(generator and discriminator) at the same like optimizing 2 functions simultaneously. However, ...
• 1
1 vote
13 views

### Using Gradient Decent For PCA Optimization [closed]

I'm trying to solve the PCA problem: For $k\in N$ some number and $X\in R(n\times d)$ where I'm trying to find $w\in R(k\times d)$ such that: $w = argmax( E(WXX^T))$ (I might be wrong with the ...
• 11
1 vote
13 views

### Gradient Descent Algorithm: For multiple local minimum which one to pick

This might be a newbie question, but it is from a newbie. If there are multiple local minimums, and the function converges at various local minima, which local minima to pick for optimization? Do we ...
• 111
1 vote
17 views

### gradient ascent vs gradient descent update rule

I'm trying to understand the differences between the update rule for stochastic gradient ascent and descent. I've read some articles and still don't understand how to calculate the update rule: ...
1 vote
29 views

### Why does SKLearn's Logistic Regression model have the same coefficients as my own model for 1 class but have different coefficients for other classes

I am currently implementing logistic regression from scratch and I'm comparing my model with SKLearn's logistic regression. Since this is just an exercise, I decided to use toy data, specifically ...
• 11
16 views

### What should the group lasso proximal operator be for no penalty?

The group lasso proximal operator is given by $$\text{prox}(\beta_j)= \left(1-\frac{\lambda}{\|\beta_j\|}\right)_+ \beta_j$$ What should this be when $\lambda = 0$ and all the input $\beta$ are 0, as ...
• 151
1 vote
31 views

### How error derivative becomes zero in gradient descent

Previous questions this & this does not answer my question Code ...
• 657
1 vote
11 views

### why we use same learning rate in the whole process of the gradient descent?

In theory, we know while we are descending to the point where the error is zero, we give big steps that are learning rate will be big. And when we are near to the error equal to zero we start giving ...
• 657
17 views

### Is gradient descent for non-parametric maximum likelihood estimation? [duplicate]

In my reading of maximum likelihood estimation, they go through samples with KNOWN distributions (e.g. binomial, poisson, etc.). I wonder how can I connect to my knowledge of machine learning. In ...
• 145
1 vote
16 views

### Gradient Descent for Multi-Level / Mixed / Hierarchical Regression Model

How would gradient descent work in a multilevel regression setting? This is fairly clear to me in a standard linear regression formulation, but haven't been able to wrap my head around parameter ...
• 11
1 vote
42 views

### Subgradient for sparse-group lasso

Sparse-group lasso is defined as $$\frac{1}{2n}\left\|y-X\beta \right\| + (1-\alpha)\lambda\sum_{l=1}^m \sqrt{p_l}\left\|\beta^{(l)} \right\|_2 + \alpha \lambda \left\| \beta\right\|_1$$ In the SGL ...
• 151
381 views

• 1,486
1 vote
563 views

### Plotting Log Likelihood

It was suggested I ask this here instead of Stack Overflow. I am trying to plot the negative log likelihood of an exponential distribution. I am not getting how I am supposed to think of it. The ...
• 11
136 views

### Optimizing logistic regression with a custom penalty using gradient descent

I'm trying to fit a logistic regression model on a certain dataset. I want to ensure the learned model is smooth, that is samples which belong to the same cluster/group according to a prior knowledge/...
• 131
349 views

### Is batching a way to avoid local minima?

That is my question: is batching one way to prevent the model from falling in a local minima? What is the difference between bach=1 and ...
• 135
72 views

### Can Gradient Descent "Bounce Around" Forever? [duplicate]

When learning about Neural Networks and Gradient Descent, we are often shown the following picture that illustrates the obstacles that can be encountered when trying to optimize the Loss Functions ...
• 6,078