Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

How is the loss function related to the derivative of a specific output neuron?

Suppose we have an output vector of three values to give a concrete example: [0, 1, 0.8]. Suppose the ground truth values are [1,1,1]. MSE loss will return about 0.35. How does the value of 0.35 ...
0 votes
0 answers
7 views

How training GANs with the method of optimizing 2 loss functions simultaneously differ from the actual alternate training?

I am training GAN in a multiobjective optimization setting where I am optimizing both the loss functions(generator and discriminator) at the same like optimizing 2 functions simultaneously. However, ...
  • 1
1 vote
0 answers
13 views

Using Gradient Decent For PCA Optimization [closed]

I'm trying to solve the PCA problem: For $k\in N$ some number and $X\in R(n\times d)$ where I'm trying to find $w\in R(k\times d)$ such that: $w = argmax( E(WXX^T))$ (I might be wrong with the ...
  • 11
1 vote
0 answers
13 views

Gradient Descent Algorithm: For multiple local minimum which one to pick

This might be a newbie question, but it is from a newbie. If there are multiple local minimums, and the function converges at various local minima, which local minima to pick for optimization? Do we ...
  • 111
1 vote
1 answer
17 views

gradient ascent vs gradient descent update rule

I'm trying to understand the differences between the update rule for stochastic gradient ascent and descent. I've read some articles and still don't understand how to calculate the update rule: ...
1 vote
1 answer
29 views

Why does SKLearn's Logistic Regression model have the same coefficients as my own model for 1 class but have different coefficients for other classes

I am currently implementing logistic regression from scratch and I'm comparing my model with SKLearn's logistic regression. Since this is just an exercise, I decided to use toy data, specifically ...
  • 11
0 votes
0 answers
16 views

What should the group lasso proximal operator be for no penalty?

The group lasso proximal operator is given by $$\text{prox}(\beta_j)= \left(1-\frac{\lambda}{\|\beta_j\|}\right)_+ \beta_j$$ What should this be when $\lambda = 0$ and all the input $\beta$ are 0, as ...
  • 151
1 vote
1 answer
31 views

How error derivative becomes zero in gradient descent

Previous questions this & this does not answer my question Code ...
  • 657
1 vote
1 answer
11 views

why we use same learning rate in the whole process of the gradient descent?

In theory, we know while we are descending to the point where the error is zero, we give big steps that are learning rate will be big. And when we are near to the error equal to zero we start giving ...
  • 657
0 votes
0 answers
17 views

Is gradient descent for non-parametric maximum likelihood estimation? [duplicate]

In my reading of maximum likelihood estimation, they go through samples with KNOWN distributions (e.g. binomial, poisson, etc.). I wonder how can I connect to my knowledge of machine learning. In ...
  • 145
1 vote
0 answers
16 views

Gradient Descent for Multi-Level / Mixed / Hierarchical Regression Model

How would gradient descent work in a multilevel regression setting? This is fairly clear to me in a standard linear regression formulation, but haven't been able to wrap my head around parameter ...
  • 11
1 vote
1 answer
42 views

Subgradient for sparse-group lasso

Sparse-group lasso is defined as $$\frac{1}{2n}\left\|y-X\beta \right\| + (1-\alpha)\lambda\sum_{l=1}^m \sqrt{p_l}\left\|\beta^{(l)} \right\|_2 + \alpha \lambda \left\| \beta\right\|_1$$ In the SGL ...
  • 151
4 votes
2 answers
381 views

ALS vs SGD in parallelization

So given the standard objective in matrix factorization for collaborative filtering of minimizing: $$ L = \sum_{u,i \in S} (r_{ui}-q_i^Tp_u)^2 + \lambda(\sum_i||q_i^2||+\sum_u||p_u^2||) $$ , where $r_{...
  • 161
1 vote
1 answer
59 views

Why do we regularize large gradients corresponding to large errors?

While reviewing some scientific blogs, I found them recommending using gradient clipping for large error gradients. However, intuitively one would think that when model predictions are completely off, ...
1 vote
2 answers
67 views

How does epoch-wise double descent occur if training error is 0?

In this paper, they talk about the existence of epoch-wise double descent. In Figure 10, you can see that, with a sufficiently large model, the test error keeps decreasing even after the training ...
0 votes
0 answers
41 views

Gradient descent / Adam converging to suboptimal solutions

I am using neural nets to find the minimum of a complex function to which I compute the mean (crit in my code). Here is my net : ...
  • 1
7 votes
1 answer
309 views

What happens when DQN gradients become too big?

I am reading these notes on slide 34 and came across strategies to prevent gradients from becoming too big in Deep Q Learning (DQN). Since, we don't usually use deep architectures in DQN, I don't ...
4 votes
1 answer
51 views

How to use the Likelihood Ratio Test and Wald Statistic when also using Cross Validation?

I am currently writing a paper for uni and stumbled across the following problem: I want to use the 10 Fold Cross Validation method to validate the results of a logistic regression, but I am unsure ...
  • 43
3 votes
1 answer
24 views

Allow gradient descent to go beyond constraint but punish for it

Disclaimer: I considered posting this on mathSE but thought maybe this is more fitting here (no pun intended). Status Quo I have a list of data points $\{x_k, y_k\}$ and a set of functions $f_i(x)$ ...
0 votes
2 answers
49 views

What will happen when we cannot find minimum through gradient descent?

Suppose that we have cost function with degree of 3. Based on gradient descent, we may have a derivative of zero in one or two points,but none of them specify the minimum of the cost function in those ...
2 votes
1 answer
20 views

How can make sure our deep neural network is differentiable

When we have a deep neural network, according to how much complicated that neural network is, how we can make sure that in each layer we can calculate the derivatives?( Is that differentiable or not). ...
2 votes
1 answer
25 views

What could go wrong if I do gradient descent by class?

For example, when training on CIFAR10, each minibatch typically contains images from all 10 classes (assuming a moderately large batch size such as 64). What could go wrong if I train on a homogeneous ...
  • 1,487
2 votes
0 answers
44 views

Natural gradients with Moore–Penrose inverse of the Fisher information matrix

I'd like to show you my rough sketch for scaling up natural gradients to deep neural networks that appears to be easy to automate just like automatic differentiation. I think there must be a flaw ...
0 votes
0 answers
6 views

Stochastic objective function in Adam [duplicate]

Relating to the question What is a stochastic objective function?. In the paper of the Adam Algorithm they also mention a Stochastic objective function when explaining the algorithm. In this context I ...
0 votes
0 answers
12 views

Averaging gradients of shared parameters

Let's say I have a machine learning model that uses the same parameter several times during a forward pass. As an example, let $f(x, y)$ be a toy model with $f(x, y) = x + x + x + y$. If I compute the ...
  • 615
1 vote
0 answers
29 views

Explanation of the derivation of the analytical gradient for a SVM?

I'm trying to understand how to derive the analytical gradient for a SVM. I know that in a SVM, the loss function is defined as follows: From this blogpost, I know the full loss for each element in ...
  • 237
1 vote
0 answers
23 views

When is RMSprop Gradient Descent better than Momentum Gradient Descent

Can you give an example where RMSprop is better than momentum? In what cases does RMSprop (i.e. scaling $d\Theta_i$) help, but Momentum does not? How is the combination of Momentum and RMSprop (i.e. ...
  • 93
2 votes
1 answer
35 views

How to make a Neural Network(NN) learn when it is an input to an non-differentiable function?

I have reinforcement/Imitation learning problem where I have to learn the Policy/controller Network ${\pi}$ for the system to be controlled. The final loss function I got involves a Black box function ...
3 votes
1 answer
41 views

Is standard gradient descent possible when we have constrained parameters?

Is it true that standard gradient descent algorithm (be it batch or mini batch or stochastic) cannot be used when we have certain constraint on parameters? If yes, why is it so? Is it because gradient ...
  • 161
2 votes
0 answers
22 views

Does XGBoost use gradient descent? [duplicate]

XGBoost is said to be based on "gradient tree boosting" in the original paper. Reading the paper and the introduction on the official website, it seems to me that the algorithm does not use ...
1 vote
0 answers
24 views

How can we find a unique (suboptimal) solution to an optimisation problem with a very large search space?

A standard linear and unconstrained optimization problem has the following form: $\max_{x} f(x)$, For example $f(x) = cos(x)+sin(2x)$, with $-1 \leq x \leq 8$. The gradient $=0$ will allow finding all ...
1 vote
1 answer
38 views

No gradient for one parameter on the first iteration of gradient descent

Say we have a dataset $D$ of 2-tuples $(x, y)$ where $y$ is the target variable and a function $f_\theta$: $$ D = \{(1, 3), (2, 5), (3, 8), (4, 6), (5, 9)\},\quad f_\theta(x) = \theta_0 + \theta_1 x. $...
0 votes
0 answers
22 views

How derivate GD with thos loss function?

Question. Consider a 2 layer neural network with no bias units for regression problems where the input feature dimension is d and the output target is one dimension. The regularized neural network ...
  • 1
0 votes
0 answers
4 views

A question on the projection step in Generic Adaptive Method Setup: $x_{t+1} = \Pi_{\mathcal{F},\sqrt{V_t}} (\hat{x}_{t+1}).$

I am reading the paper "ON THE CONVERGENCE OF ADAM AND BEYOND". In this paper, they proposed the following framework of adaptive methods. I was confused on the last step: $x_{t+1} = \Pi_{\...
  • 101
1 vote
2 answers
76 views

Why use transpose of nabla in gradient descent

For gradient descent we have the formula: $f(x_{k}+d_{k})\approx f(x_{k}) + \nabla f(x_{k})^T d_{k} $ What I don't understand is, why we use the transpose of nabla and not just nabla.
  • 13
1 vote
0 answers
74 views

Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network [closed]

How would I go about deriving gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward propagation expressed as: $$ F(...
  • 11
1 vote
1 answer
36 views

Is gradient descent the only way to find the weights in logistic regression?

This post: When is logistic regression solved in closed form? describes that we must use nonlinear optimization methods to find the parameter estimates for logistic regression models. Does gradient ...
  • 55
0 votes
0 answers
39 views

Projected Gradient Descent for neural networks

One of the most widely used optimization methods for neural networks is (stochastic) gradient descent. When encountering constrained problems, a standard modification of gradient descent consists of ...
  • 105
0 votes
1 answer
21 views

How are samples selected when batching?

Suppose after being randomized samples in a dataset can be written $\{x_i\}_{i = 0}^n$ and are put into a batch for mini-batch gradient descent. Are the samples drawn randomly into a batch, or are the ...
  • 121
1 vote
1 answer
128 views

If a regression problem is ill-conditioned, does that mean we cannot perform SGD? What happens if we do?

By ill-conditioned regression problem, I mean that the feature matrix $X$ is not full rank. For example, X contains two or more columns highly correlated. If that's the case, $X^T\cdot X$ is not ...
  • 185
1 vote
1 answer
69 views

Solve Local Minima Problem through Averaging

I am using neural networks (created in R: neuralnet) to predict county-level food insecurity. I want to use Olden's connection weight approach to analyze relative ...
  • 29
2 votes
1 answer
181 views

Why not use alternating minimization for training neural networks?

Neural networks are usually trained by calculating gradients using backprop and then performing gradient descent. I am not sure what are difficulties in training a neural network using alternating ...
  • 175
3 votes
1 answer
109 views

Unexpected weights in Gradient descent algorithm (linear classification) in python

I am attempting to implement a back propagation algorithm that can efficevley read from a file of features and there targets and predict there outputs correctly, however for sake of testing I am hard ...
1 vote
2 answers
276 views

What is the difference between gradient descent and batch gradient descent? [duplicate]

It seems that batch gradient descent is the traditional gradient descent, except that the objective function is in the form of summation?
2 votes
0 answers
67 views

Diverging Gradient Boosting Optimization with custom loss

I have a supervised learning problem (regression) with $m$ input features $X=(x_1, ..., x_m)$ with output $y$. I want to predict a "multiplicative correction factor" $\hat \alpha(X)$ such ...
  • 1,486
1 vote
1 answer
563 views

Plotting Log Likelihood

It was suggested I ask this here instead of Stack Overflow. I am trying to plot the negative log likelihood of an exponential distribution. I am not getting how I am supposed to think of it. The ...
  • 11
2 votes
1 answer
136 views

Optimizing logistic regression with a custom penalty using gradient descent

I'm trying to fit a logistic regression model on a certain dataset. I want to ensure the learned model is smooth, that is samples which belong to the same cluster/group according to a prior knowledge/...
  • 131
2 votes
1 answer
349 views

Is batching a way to avoid local minima?

That is my question: is batching one way to prevent the model from falling in a local minima? What is the difference between bach=1 and ...
  • 135
2 votes
0 answers
72 views

Can Gradient Descent "Bounce Around" Forever? [duplicate]

When learning about Neural Networks and Gradient Descent, we are often shown the following picture that illustrates the obstacles that can be encountered when trying to optimize the Loss Functions ...
  • 6,078
2 votes
1 answer
295 views

Does Stochastic Gradient Descent Converge on "some" Non-Convex Functions?

I am interested in better understanding the theoretical behavior (i.e. strength of convergence) of Stochastic Gradient Descent on Non-Convex Functions. We are constantly reminded that Stochastic ...
  • 6,078

1
2 3 4 5
19