Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

How derivate GD with thos loss function?

Question. Consider a 2 layer neural network with no bias units for regression problems where the input feature dimension is d and the output target is one dimension. The regularized neural network ...
user avatar
  • 1
0 votes
0 answers
3 views

A question on the projection step in Generic Adaptive Method Setup: $x_{t+1} = \Pi_{\mathcal{F},\sqrt{V_t}} (\hat{x}_{t+1}).$

I am reading the paper "ON THE CONVERGENCE OF ADAM AND BEYOND". In this paper, they proposed the following framework of adaptive methods. I was confused on the last step: $x_{t+1} = \Pi_{\...
user avatar
  • 101
1 vote
2 answers
32 views

Why use transpose of nabla in gradient descent

For gradient descent we have the formula: $f(x_{k}+d_{k})\approx f(x_{k}) + \nabla f(x_{k})^T d_{k} $ What I don't understand is, why we use the transpose of nabla and not just nabla.
user avatar
  • 13
1 vote
0 answers
51 views

Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network [closed]

How would I go about deriving gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward propagation expressed as: $$ F(...
user avatar
  • 11
1 vote
1 answer
28 views

Is gradient descent the only way to find the weights in logistic regression?

This post: When is logistic regression solved in closed form? describes that we must use nonlinear optimization methods to find the parameter estimates for logistic regression models. Does gradient ...
user avatar
  • 55
0 votes
0 answers
15 views

Projected Gradient Descent for neural networks

One of the most widely used optimization methods for neural networks is (stochastic) gradient descent. When encountering constrained problems, a standard modification of gradient descent consists of ...
user avatar
  • 105
0 votes
1 answer
10 views

How are samples selected when batching?

Suppose after being randomized samples in a dataset can be written $\{x_i\}_{i = 0}^n$ and are put into a batch for mini-batch gradient descent. Are the samples drawn randomly into a batch, or are the ...
user avatar
  • 121
0 votes
1 answer
29 views

If a regression problem is ill-conditioned, does that mean we cannot perform SGD? What happens if we do?

By ill-conditioned regression problem, I mean that the feature matrix $X$ is not full rank. For example, X contains two or more columns highly correlated. If that's the case, $X^T\cdot X$ is not ...
user avatar
  • 143
1 vote
1 answer
52 views

Solve Local Minima Problem through Averaging

I am using neural networks (created in R: neuralnet) to predict county-level food insecurity. I want to use Olden's connection weight approach to analyze relative ...
user avatar
  • 29
2 votes
1 answer
126 views

Why not use alternating minimization for training neural networks?

Neural networks are usually trained by calculating gradients using backprop and then performing gradient descent. I am not sure what are difficulties in training a neural network using alternating ...
user avatar
  • 185
3 votes
1 answer
38 views

Unexpected weights in Gradient descent algorithm (linear classification) in python

I am attempting to implement a back propagation algorithm that can efficevley read from a file of features and there targets and predict there outputs correctly, however for sake of testing I am hard ...
user avatar
1 vote
2 answers
62 views

What is the difference between gradient descent and batch gradient descent? [duplicate]

It seems that batch gradient descent is the traditional gradient descent, except that the objective function is in the form of summation?
user avatar
2 votes
0 answers
42 views

Diverging Gradient Boosting Optimization with custom loss

I have a supervised learning problem (regression) with $m$ input features $X=(x_1, ..., x_m)$ with output $y$. I want to predict a "multiplicative correction factor" $\hat \alpha(X)$ such ...
user avatar
  • 1,486
1 vote
1 answer
156 views

Plotting Log Likelihood

It was suggested I ask this here instead of Stack Overflow. I am trying to plot the negative log likelihood of an exponential distribution. I am not getting how I am supposed to think of it. The ...
user avatar
  • 11
2 votes
1 answer
71 views

Optimizing logistic regression with a custom penalty using gradient descent

I'm trying to fit a logistic regression model on a certain dataset. I want to ensure the learned model is smooth, that is samples which belong to the same cluster/group according to a prior knowledge/...
user avatar
  • 131
2 votes
1 answer
210 views

Is batching a way to avoid local minima?

That is my question: is batching one way to prevent the model from falling in a local minima? What is the difference between bach=1 and ...
user avatar
  • 135
1 vote
0 answers
50 views

Can Gradient Descent "Bounce Around" Forever? [duplicate]

When learning about Neural Networks and Gradient Descent, we are often shown the following picture that illustrates the obstacles that can be encountered when trying to optimize the Loss Functions ...
user avatar
  • 5,650
2 votes
1 answer
135 views

Does Stochastic Gradient Descent Converge on "some" Non-Convex Functions?

I am interested in better understanding the theoretical behavior (i.e. strength of convergence) of Stochastic Gradient Descent on Non-Convex Functions. We are constantly reminded that Stochastic ...
user avatar
  • 5,650
1 vote
0 answers
41 views

Convergence rates of Quasi-Newton methods vs. Gradient Descent

I am interested in learning about if Optimization Algorithms that use both First Derivative Information and Second Derivative Information have any advantages (e.g. strength of Convergence) when ...
user avatar
  • 5,650
0 votes
0 answers
36 views

Can we conclude that Stochastic Gradient Descent is a "superior" algorithm that Gradient Descent?

On a very informal level, if we were to compare the (classical) Gradient Descent Algorithm to the Stochastic Gradient Descent Algorithm, the first thing that comes to mind is: Gradient Descent can be ...
user avatar
  • 5,650
0 votes
1 answer
69 views

Examples in Machine Learning with Non-Differentiable Objective Functions

I was reading the following lecture notes on Gradient Descent and came across the following note: Supposedly, there are some instances in machine learning where the objective function is non-...
user avatar
  • 5,650
0 votes
0 answers
36 views

Proof of Stochastic Gradient Descent

When it comes to the classical Gradient Descent Algorithm, for optimizing Convex Functions - there is a standard proof that demonstrates the convergence of this algorithm provided certain conditions ...
user avatar
  • 5,650
0 votes
0 answers
101 views

No Free Lunch Theorem and Random Search

Recently, I posted a question about a very interesting property of the Random Search Algorithm (The "Amazing Hidden Power" of Random Search?), in particular - in the context of function ...
user avatar
  • 5,650
0 votes
1 answer
64 views

Why do we always illustrate and depict the Loss Functions of Neural Networks as Non-Convex?

Why do we always illustrate and depict the Loss Functions of Neural Networks as Non-Convex? By doing a quick Google Images Search of "Loss Functions for Neural Networks" - we are generally ...
user avatar
  • 5,650
-1 votes
2 answers
83 views

Why are the "Loss Functions" being Optimized in most Statistical/Machine Learning Problems usually "Quadratic"? [duplicate]

Why are the "Loss Functions" being Optimized in most Statistical/Machine Learning Problems usually "Quadratic"? Using very basic logic, in statistics/machine learning we are trying ...
user avatar
  • 5,650
0 votes
0 answers
40 views

How to Initialize Values for Optimization Algorithms?

I am interested using Gradient Based Optimization Algorithms (e.g. BFGS) for optimizing the Rastrign Function (over the range of (0,0) to (5,5)): ...
user avatar
  • 5,650
0 votes
0 answers
37 views

Benefits of Expectation Maximization for Mixture Models

What are the benefits of using expectation maximization for mixture models vs. direct maximization of the marginal likelihoods? Analytic maximization step In case of Gaussian mixtures the benefit is ...
user avatar
  • 1,578
31 votes
11 answers
7k views

The "Amazing Hidden Power" of Random Search?

I have the following question that compares random search optimization with gradient descent optimization: Based on the (amazing) answer provided over here Optimization when Cost Function Slow to ...
user avatar
  • 5,650
0 votes
1 answer
466 views

Why does Adam optimizer seems to prevail over Nadam optimizer? [closed]

I have been studying the way Adam optimizer works, and how it combines both RMSProp and Momentum optimizers. So the following question arises: Why not combine Nesterov Accelerated Gradient together ...
user avatar
-1 votes
1 answer
62 views

Are ALL "Loss Functions" in Statistical and Machine Learning Models Fundamentally "Noisy"?

I am trying to better understand the meaning of "noise" with regards to function optimization. Up until now, I always thought of "noise" from a signal processing standpoint: for ...
user avatar
  • 5,650
0 votes
0 answers
29 views

Theoretical Results of Optimization Algorithms on Non-Convex and "Noisy" Functions

Have any major results been established on the use of gradient descent for optimizing non-convex and noisy functions? It seems like the majority of the desirable properties of gradient based ...
user avatar
  • 5,650
0 votes
0 answers
10 views

Comparing Convergence Results Between Gradient Descent and Stochastic Gradient Descent

In general, I have heard that Gradient Descent has better convergence compared to Stochastic Gradient Descent - but Stochastic Gradient Descent is more computationally effective with higher ...
user avatar
  • 5,650
1 vote
0 answers
64 views

Why is Gradient Descent considered the "go to" Algorithm for Optimizing Neural Networks? [duplicate]

Does anyone know why "Gradient Descent" Algorithms are considered as the "go to" choice of optimization algorithms for neural networks? In particular, I am interested in knowing ...
user avatar
  • 5,650
2 votes
1 answer
262 views

Why do the derivatives of a function lead towards the extremum of the function?

Is there some theorem in mathematics that formalizes the idea that "for some function, at a given point, moving in the negative direction of the gradient leads you to some (local) extremum point&...
user avatar
  • 5,650
1 vote
2 answers
131 views

Criticism of Random Search Methods in Optimization and Machine Learning

I had the following question relating to "Random Search Methods" in Optimization and Machine Learning - in short, are there theoretical results which show the obvious idea: Why are "...
user avatar
  • 5,650
1 vote
1 answer
29 views

Limit of Momentum Update Equation

I am self-studying on optimization algorithm on https://d2l.ai/chapter_optimization/momentum.html and couldn't get my head around some derivation: Instead of the standard gradient descent update ...
user avatar
1 vote
0 answers
14 views

Neural network cost converges to 1.0 through backpropagation [duplicate]

I let my NN train overnight on a subset of the MNIST training data (2000 distinct digits) and the cost function for the same subset ended up converging quickly to 1.0 after about 10 gradient descents. ...
user avatar
1 vote
0 answers
27 views

Is the difference between SGD and GD really in the summation? [duplicate]

Model: $\min _{\mathbf{w}} \frac{\lambda}{2}\|\mathbf{w}\|^{2}+\frac{1}{m} \sum_{(\mathbf{x}, y) \in S} \ell(\mathbf{w} ;(\phi(\mathbf{x}), y))$ \begin{aligned} &\text { INPUT: } S, \lambda, T \\ ...
user avatar
  • 224
0 votes
0 answers
58 views

Levenberg-Marquardt made scale invariant with diagonal matrix?

From the paper Improvements to the Levenberg-Marquardt algorithm for nonlinear least-squares minimization We now describe how to choose an effective damping matrix $D^TD$. Levenberg originally ...
user avatar
0 votes
1 answer
35 views

Can someone explain Stein's method/discrepancy in a way that makes sense?

I have been wanting to understand this paper in a deeper way for a long time Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm But everytime I read about Stein's ...
user avatar
  • 619
4 votes
1 answer
80 views

How does early stopping act as a regulizer?

Reading the book "Deep Learning" by Goodfellow et al., in section 7.8 (for ref. Deep Learning - Chapter 7), I came across a demonstration of why early stopping can be interpreted as a ...
user avatar
0 votes
0 answers
6 views

is RPROP (resilient backpropagation) garanteed to converge?

I use RPROP, RPROP+, iRProp+ algorithms for gradient descent for an error function that is differentiable everywhere. Classicaly, my configuration parameters are 1.2 for increasing a step-size in case ...
user avatar
1 vote
1 answer
63 views

Gradient descent implementation of logistic regression

Objective Seeking for help, advise why the gradient descent implementation does not work below. Background Working on the task below to implement the logistic regression. Gradient descent Derived the ...
user avatar
  • 755
1 vote
1 answer
138 views

Is backpropagation a fancy way of saying "calculate gradient by taking partial derivative w.r.t. all x's"

I understanding how gradient is calculated in the usual context--it is just taking partial derivative w.r.t. each element of the X vector. Say a function $f$ has $n$ independent variables, denoted by $...
user avatar
4 votes
1 answer
265 views

Why not solve SVM with gradient descent instead of quadratic programming?

How is SVM optimization implemented in packages like Scikit-Learn? Clearly, SVM is a quadratic programming problem but why not just use gradient descent to update the parameters? Is it because we want ...
user avatar
  • 118
0 votes
0 answers
29 views

Is this loss function convex?

I am working on a small project to understand gradient descent. I am trying to model noisy cubic data with the ideal cubic polynomial. I am using linear regression with the least-squares method with ...
user avatar
2 votes
0 answers
64 views

Validation loss falls but train loss remains constant? [closed]

My validation loss (left) falls to near 0, while my training loss (right) remains basically unchanged (gradient step is on the abscissa). This is the opposite of the typical error in which train loss ...
user avatar
6 votes
2 answers
505 views

matrix-calculus - Understanding numerator/denominator layouts

Consider the following machine-learning model: Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of training-examples. While performing reverse-mode ...
user avatar
  • 206
0 votes
0 answers
16 views

Fit a Gaussian function (not a Gaussian distribution) to data

I am trying to train a Gaussian function. NOTE: Since we're on a Q&A site with one of its focuses being Statistics, it is worth emphasizing that not all Gaussian functions are normal ...
user avatar
  • 3,341
0 votes
0 answers
17 views

Adding New Regularization Terms to Closed Form LSE

I am trying to align the input space X to the output space Y by using least squares method in a closed form solution. To do that I use svd for finding the rotation matrix (W). And I find a solution ...
user avatar

1
2 3 4 5
19