Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

870 questions
Filter by
Sorted by
Tagged with
19 views

I am a beginner to neural networks and I am writing a report summarising on the causes and solutions to the vanishing gradient problem. From what I have read, the 2 main causes are the repeated ...
21 views

### Is it possible to learn with batch size = 1?

Due to OOM error, I can only set the batch size to be 2 or 1. Is it possible to learn with such a low batch size? Thanks!
20 views

### Multi-variate Linear Regression using Gradient Descent [closed]

I've been trying to write a code for Linear Regression using Gradient Descent formula: ...
36 views

### Quantitatively define "small gradient" when checking convergence

When checking if a gradient descent (GD) has reached a minimum, it's a common practice to check the gradient of the cost function at the final iterate (also one might check if the Hessian is positive ...
32 views

### Gradient descent finds local minima for a problem that can be formulated as a convex problem

I am trying to find $$\min_W \|Y-XW \|_F^2$$ $$s.t. \exists ij, W_{ij}\geq0$$ where X is input data and Y is the output data we try to fit to. This is a convex optimization problem that can be ...
8 views

### Vanishing gradient problem and choice of cost function

I am reading chapter 6.2.7 vanishing gradient problem in the book Ovidiu Calin - Deep Learning Architectures - A Mathematical Approach. On page 187 the author mentioned one of the causes, i.e. the ...
14 views

### Minimizing the cost function using Gradient descent and chain rule

In the above picture, RV and RV hat are the actual and predicted values of RV (Realized volatility). Lambda hat ought to minimize the cost function (RHS of equation 1). In order to estimate lambda hat,...
47 views

### Neural Networks: How to get the gradient vector for the xOr problem?

I'm reading about neural networks, but the material I find is sometimes very abstract or just copies of something. Well, when considering the $xOr$ problem, I have a network in the following structure ...
34 views

### Why are non-linear activation functions required in multilayer perceptron classification? [duplicate]

Solution: for some reason, I had forgotten that the non-linear activation function is applied at every layer of the neural network, not just at the output layer. Hopefully to others reading my ...
38 views

### Why do we use gradient descent on loss functions that change depending on the input?

I've noticed that some loss functions (in the context of measuring accuracy of some model on a dataset) "depend on the input" in that different inputs will end up being evaluated differently ...
16 views

### Mathematical formalism of Gradient Boosting Decision Trees (GBDT) algorithms

I'm trying to better figure out some formalism behind the Gradient Boosting Decision Trees (GBDT) algorithms. Given a dataset $\mathcal{D}$ and a loss function $L : \mathbb{R}^2 \rightarrow \mathbb{R}$...
432 views

### Which approaches exist for optimization in machine learning?

From this blog post: For any Optimization problem with respect to Machine Learning, there can be either a numerical approach or an analytical approach. The numerical problems are Deterministic, ...
359 views

I have already gone through the post and this post, but they didn't clear my doubt. Let us say if I have a deep neural network like (having more layers about 50): Now, my question is: If I'm using an ...
40 views

### Why isn't my gradient descent code converging to solution for GB2 probability distribution?

I'm running gradient descent code in R on an $n$=10,000 test dataset simulating insurance claims records that follow the Generalized Beta of the 2nd Kind ...