Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

905 questions
Filter by
Sorted by
Tagged with
19 views

### How derivate GD with thos loss function?

Question. Consider a 2 layer neural network with no bias units for regression problems where the input feature dimension is d and the output target is one dimension. The regularized neural network ...
• 1
3 views

I am reading the paper "ON THE CONVERGENCE OF ADAM AND BEYOND". In this paper, they proposed the following framework of adaptive methods. I was confused on the last step: $x_{t+1} = \Pi_{\... • 101 1 vote 2 answers 32 views ### Why use transpose of nabla in gradient descent For gradient descent we have the formula:$f(x_{k}+d_{k})\approx f(x_{k}) + \nabla f(x_{k})^T d_{k} $What I don't understand is, why we use the transpose of nabla and not just nabla. • 13 1 vote 0 answers 51 views ### Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network [closed] How would I go about deriving gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward propagation expressed as:$$F(... • 11 1 vote 1 answer 28 views ### Is gradient descent the only way to find the weights in logistic regression? This post: When is logistic regression solved in closed form? describes that we must use nonlinear optimization methods to find the parameter estimates for logistic regression models. Does gradient ... • 55 0 votes 0 answers 15 views ### Projected Gradient Descent for neural networks One of the most widely used optimization methods for neural networks is (stochastic) gradient descent. When encountering constrained problems, a standard modification of gradient descent consists of ... • 105 0 votes 1 answer 10 views ### How are samples selected when batching? Suppose after being randomized samples in a dataset can be written$\{x_i\}_{i = 0}^n$and are put into a batch for mini-batch gradient descent. Are the samples drawn randomly into a batch, or are the ... • 121 0 votes 1 answer 29 views ### If a regression problem is ill-conditioned, does that mean we cannot perform SGD? What happens if we do? By ill-conditioned regression problem, I mean that the feature matrix$X$is not full rank. For example, X contains two or more columns highly correlated. If that's the case,$X^T\cdot X$is not ... • 143 1 vote 1 answer 52 views ### Solve Local Minima Problem through Averaging I am using neural networks (created in R: neuralnet) to predict county-level food insecurity. I want to use Olden's connection weight approach to analyze relative ... • 29 2 votes 1 answer 126 views ### Why not use alternating minimization for training neural networks? Neural networks are usually trained by calculating gradients using backprop and then performing gradient descent. I am not sure what are difficulties in training a neural network using alternating ... • 185 3 votes 1 answer 38 views ### Unexpected weights in Gradient descent algorithm (linear classification) in python I am attempting to implement a back propagation algorithm that can efficevley read from a file of features and there targets and predict there outputs correctly, however for sake of testing I am hard ... 1 vote 2 answers 62 views ### What is the difference between gradient descent and batch gradient descent? [duplicate] It seems that batch gradient descent is the traditional gradient descent, except that the objective function is in the form of summation? 2 votes 0 answers 42 views ### Diverging Gradient Boosting Optimization with custom loss I have a supervised learning problem (regression) with$m$input features$X=(x_1, ..., x_m)$with output$y$. I want to predict a "multiplicative correction factor"$\hat \alpha(X)$such ... • 1,486 1 vote 1 answer 156 views ### Plotting Log Likelihood It was suggested I ask this here instead of Stack Overflow. I am trying to plot the negative log likelihood of an exponential distribution. I am not getting how I am supposed to think of it. The ... • 11 2 votes 1 answer 71 views ### Optimizing logistic regression with a custom penalty using gradient descent I'm trying to fit a logistic regression model on a certain dataset. I want to ensure the learned model is smooth, that is samples which belong to the same cluster/group according to a prior knowledge/... • 131 2 votes 1 answer 210 views ### Is batching a way to avoid local minima? That is my question: is batching one way to prevent the model from falling in a local minima? What is the difference between bach=1 and ... • 135 1 vote 0 answers 50 views ### Can Gradient Descent "Bounce Around" Forever? [duplicate] When learning about Neural Networks and Gradient Descent, we are often shown the following picture that illustrates the obstacles that can be encountered when trying to optimize the Loss Functions ... • 5,650 2 votes 1 answer 135 views ### Does Stochastic Gradient Descent Converge on "some" Non-Convex Functions? I am interested in better understanding the theoretical behavior (i.e. strength of convergence) of Stochastic Gradient Descent on Non-Convex Functions. We are constantly reminded that Stochastic ... • 5,650 1 vote 0 answers 41 views ### Convergence rates of Quasi-Newton methods vs. Gradient Descent I am interested in learning about if Optimization Algorithms that use both First Derivative Information and Second Derivative Information have any advantages (e.g. strength of Convergence) when ... • 5,650 0 votes 0 answers 36 views ### Can we conclude that Stochastic Gradient Descent is a "superior" algorithm that Gradient Descent? On a very informal level, if we were to compare the (classical) Gradient Descent Algorithm to the Stochastic Gradient Descent Algorithm, the first thing that comes to mind is: Gradient Descent can be ... • 5,650 0 votes 1 answer 69 views ### Examples in Machine Learning with Non-Differentiable Objective Functions I was reading the following lecture notes on Gradient Descent and came across the following note: Supposedly, there are some instances in machine learning where the objective function is non-... • 5,650 0 votes 0 answers 36 views ### Proof of Stochastic Gradient Descent When it comes to the classical Gradient Descent Algorithm, for optimizing Convex Functions - there is a standard proof that demonstrates the convergence of this algorithm provided certain conditions ... • 5,650 0 votes 0 answers 101 views ### No Free Lunch Theorem and Random Search Recently, I posted a question about a very interesting property of the Random Search Algorithm (The "Amazing Hidden Power" of Random Search?), in particular - in the context of function ... • 5,650 0 votes 1 answer 64 views ### Why do we always illustrate and depict the Loss Functions of Neural Networks as Non-Convex? Why do we always illustrate and depict the Loss Functions of Neural Networks as Non-Convex? By doing a quick Google Images Search of "Loss Functions for Neural Networks" - we are generally ... • 5,650 -1 votes 2 answers 83 views ### Why are the "Loss Functions" being Optimized in most Statistical/Machine Learning Problems usually "Quadratic"? [duplicate] Why are the "Loss Functions" being Optimized in most Statistical/Machine Learning Problems usually "Quadratic"? Using very basic logic, in statistics/machine learning we are trying ... • 5,650 0 votes 0 answers 40 views ### How to Initialize Values for Optimization Algorithms? I am interested using Gradient Based Optimization Algorithms (e.g. BFGS) for optimizing the Rastrign Function (over the range of (0,0) to (5,5)): ... • 5,650 0 votes 0 answers 37 views ### Benefits of Expectation Maximization for Mixture Models What are the benefits of using expectation maximization for mixture models vs. direct maximization of the marginal likelihoods? Analytic maximization step In case of Gaussian mixtures the benefit is ... • 1,578 31 votes 11 answers 7k views ### The "Amazing Hidden Power" of Random Search? I have the following question that compares random search optimization with gradient descent optimization: Based on the (amazing) answer provided over here Optimization when Cost Function Slow to ... • 5,650 0 votes 1 answer 466 views ### Why does Adam optimizer seems to prevail over Nadam optimizer? [closed] I have been studying the way Adam optimizer works, and how it combines both RMSProp and Momentum optimizers. So the following question arises: Why not combine Nesterov Accelerated Gradient together ... -1 votes 1 answer 62 views ### Are ALL "Loss Functions" in Statistical and Machine Learning Models Fundamentally "Noisy"? I am trying to better understand the meaning of "noise" with regards to function optimization. Up until now, I always thought of "noise" from a signal processing standpoint: for ... • 5,650 0 votes 0 answers 29 views ### Theoretical Results of Optimization Algorithms on Non-Convex and "Noisy" Functions Have any major results been established on the use of gradient descent for optimizing non-convex and noisy functions? It seems like the majority of the desirable properties of gradient based ... • 5,650 0 votes 0 answers 10 views ### Comparing Convergence Results Between Gradient Descent and Stochastic Gradient Descent In general, I have heard that Gradient Descent has better convergence compared to Stochastic Gradient Descent - but Stochastic Gradient Descent is more computationally effective with higher ... • 5,650 1 vote 0 answers 64 views ### Why is Gradient Descent considered the "go to" Algorithm for Optimizing Neural Networks? [duplicate] Does anyone know why "Gradient Descent" Algorithms are considered as the "go to" choice of optimization algorithms for neural networks? In particular, I am interested in knowing ... • 5,650 2 votes 1 answer 262 views ### Why do the derivatives of a function lead towards the extremum of the function? Is there some theorem in mathematics that formalizes the idea that "for some function, at a given point, moving in the negative direction of the gradient leads you to some (local) extremum point&... • 5,650 1 vote 2 answers 131 views ### Criticism of Random Search Methods in Optimization and Machine Learning I had the following question relating to "Random Search Methods" in Optimization and Machine Learning - in short, are there theoretical results which show the obvious idea: Why are "... • 5,650 1 vote 1 answer 29 views ### Limit of Momentum Update Equation I am self-studying on optimization algorithm on https://d2l.ai/chapter_optimization/momentum.html and couldn't get my head around some derivation: Instead of the standard gradient descent update ... • 113 1 vote 0 answers 14 views ### Neural network cost converges to 1.0 through backpropagation [duplicate] I let my NN train overnight on a subset of the MNIST training data (2000 distinct digits) and the cost function for the same subset ended up converging quickly to 1.0 after about 10 gradient descents. ... 1 vote 0 answers 27 views ### Is the difference between SGD and GD really in the summation? [duplicate] Model:$\min _{\mathbf{w}} \frac{\lambda}{2}\|\mathbf{w}\|^{2}+\frac{1}{m} \sum_{(\mathbf{x}, y) \in S} \ell(\mathbf{w} ;(\phi(\mathbf{x}), y))\begin{aligned} &\text { INPUT: } S, \lambda, T \\ ... • 224 0 votes 0 answers 58 views ### Levenberg-Marquardt made scale invariant with diagonal matrix? From the paper Improvements to the Levenberg-Marquardt algorithm for nonlinear least-squares minimization We now describe how to choose an effective damping matrixD^TD$. Levenberg originally ... • 101 0 votes 1 answer 35 views ### Can someone explain Stein's method/discrepancy in a way that makes sense? I have been wanting to understand this paper in a deeper way for a long time Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm But everytime I read about Stein's ... • 619 4 votes 1 answer 80 views ### How does early stopping act as a regulizer? Reading the book "Deep Learning" by Goodfellow et al., in section 7.8 (for ref. Deep Learning - Chapter 7), I came across a demonstration of why early stopping can be interpreted as a ... 0 votes 0 answers 6 views ### is RPROP (resilient backpropagation) garanteed to converge? I use RPROP, RPROP+, iRProp+ algorithms for gradient descent for an error function that is differentiable everywhere. Classicaly, my configuration parameters are 1.2 for increasing a step-size in case ... 1 vote 1 answer 63 views ### Gradient descent implementation of logistic regression Objective Seeking for help, advise why the gradient descent implementation does not work below. Background Working on the task below to implement the logistic regression. Gradient descent Derived the ... • 755 1 vote 1 answer 138 views ### Is backpropagation a fancy way of saying "calculate gradient by taking partial derivative w.r.t. all x's" I understanding how gradient is calculated in the usual context--it is just taking partial derivative w.r.t. each element of the X vector. Say a function$f$has$n$independent variables, denoted by$...
265 views

How is SVM optimization implemented in packages like Scikit-Learn? Clearly, SVM is a quadratic programming problem but why not just use gradient descent to update the parameters? Is it because we want ...
• 118
29 views

### Is this loss function convex?

I am working on a small project to understand gradient descent. I am trying to model noisy cubic data with the ideal cubic polynomial. I am using linear regression with the least-squares method with ...
• 113
64 views

### Validation loss falls but train loss remains constant? [closed]

My validation loss (left) falls to near 0, while my training loss (right) remains basically unchanged (gradient step is on the abscissa). This is the opposite of the typical error in which train loss ...
505 views

### matrix-calculus - Understanding numerator/denominator layouts

Consider the following machine-learning model: Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of training-examples. While performing reverse-mode ...
• 206