Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

Is saddle point a cause for the vanishing gradient problem

I am a beginner to neural networks and I am writing a report summarising on the causes and solutions to the vanishing gradient problem. From what I have read, the 2 main causes are the repeated ...
1
vote
1answer
21 views

Is it possible to learn with batch size = 1?

Due to OOM error, I can only set the batch size to be 2 or 1. Is it possible to learn with such a low batch size? Thanks!
0
votes
0answers
20 views

Multi-variate Linear Regression using Gradient Descent [closed]

I've been trying to write a code for Linear Regression using Gradient Descent formula: ...
2
votes
0answers
36 views

Quantitatively define "small gradient" when checking convergence

When checking if a gradient descent (GD) has reached a minimum, it's a common practice to check the gradient of the cost function at the final iterate (also one might check if the Hessian is positive ...
2
votes
0answers
32 views

Gradient descent finds local minima for a problem that can be formulated as a convex problem

I am trying to find $$ \min_W \|Y-XW \|_F^2$$ $$s.t. \exists ij, W_{ij}\geq0 $$ where X is input data and Y is the output data we try to fit to. This is a convex optimization problem that can be ...
1
vote
0answers
8 views

Vanishing gradient problem and choice of cost function

I am reading chapter 6.2.7 vanishing gradient problem in the book Ovidiu Calin - Deep Learning Architectures - A Mathematical Approach. On page 187 the author mentioned one of the causes, i.e. the ...
0
votes
0answers
14 views

Minimizing the cost function using Gradient descent and chain rule

In the above picture, RV and RV hat are the actual and predicted values of RV (Realized volatility). Lambda hat ought to minimize the cost function (RHS of equation 1). In order to estimate lambda hat,...
3
votes
1answer
47 views

Neural Networks: How to get the gradient vector for the xOr problem?

I'm reading about neural networks, but the material I find is sometimes very abstract or just copies of something. Well, when considering the $xOr$ problem, I have a network in the following structure ...
0
votes
2answers
34 views

Why are non-linear activation functions required in multilayer perceptron classification? [duplicate]

Solution: for some reason, I had forgotten that the non-linear activation function is applied at every layer of the neural network, not just at the output layer. Hopefully to others reading my ...
0
votes
0answers
38 views

Why do we use gradient descent on loss functions that change depending on the input?

I've noticed that some loss functions (in the context of measuring accuracy of some model on a dataset) "depend on the input" in that different inputs will end up being evaluated differently ...
0
votes
0answers
16 views

Mathematical formalism of Gradient Boosting Decision Trees (GBDT) algorithms

I'm trying to better figure out some formalism behind the Gradient Boosting Decision Trees (GBDT) algorithms. Given a dataset $\mathcal{D}$ and a loss function $L : \mathbb{R}^2 \rightarrow \mathbb{R}$...
4
votes
1answer
432 views

Which approaches exist for optimization in machine learning?

From this blog post: For any Optimization problem with respect to Machine Learning, there can be either a numerical approach or an analytical approach. The numerical problems are Deterministic, ...
2
votes
1answer
359 views

Misconception about ReLu

I have already gone through the post and this post, but they didn't clear my doubt. Let us say if I have a deep neural network like (having more layers about 50): Now, my question is: If I'm using an ...
0
votes
0answers
40 views

Why isn't my gradient descent code converging to solution for GB2 probability distribution?

I'm running gradient descent code in R on an $n$=10,000 test dataset simulating insurance claims records that follow the Generalized Beta of the 2nd Kind ...
1
vote
2answers
71 views

How does gradient descent help SVM learn a linearly separable hyperplane?

So I see the Perceptron Algorithm applied to learning an SVM, where $\theta$ is the normal vector to the linearly separating hyperplane. How does the update $$\theta^{t+1}\leftarrow\theta^t+\alpha ...
0
votes
1answer
38 views

For MSE equation does order of $y$ and $\hat{y}$ in the residual $(y-\hat{y})$ matter?

So the equation for MSE is $\frac{1}{2N}\sum(y-\hat{y})^2$. If you switch the order as in $\frac{1}{2N}\sum(\hat{y} - y)^2$ does that affect anything? The only thing I think it potentially effects is ...
0
votes
0answers
40 views

When are very small learning rates useful?

I just wondered if there are cases where small or very small learning rates in gradient descent based optimization are useful? A large learning rate allows the model to explore a much larger portion ...
0
votes
0answers
36 views

Why does the loss of a neural net flat-line and then suddenly drop?

The loss graph for my neural net looks like this: Blue is validation data loss and green is the training data loss. As you can see, the loss remains almost flat for the first 600 epochs and then it ...
0
votes
0answers
27 views

Gradient Descent Algorithm for Interdependent parameters

Suppose I have $n$ data points ($X_i$,$y_i$) where $X_i$ is a vector and $y_i$ is a scalar, $1 \le i \le n$. By defining $\hat{\boldsymbol{Y}} = \boldsymbol{\Theta} \boldsymbol{X} + \boldsymbol{b}$ ...
0
votes
0answers
42 views

Why do all activation functions have positive slope?

I am wondering why all the common activation functions tend to increase with x (or stay flat like ReLU). I have not come across any that are inversely proportional to x, or that have some other shape. ...
0
votes
1answer
43 views

What are the variations of Expectation Maximization?

To explain my question better, I will use this analogy: In the case of the Gradient-Descent method, we have multiple variations/expansions for the main algorithm, like stochastic gradient descent (SGD)...
1
vote
0answers
12 views

Reinforcement Learning: weights turn NaN [closed]

I'm currently implementing Q-Learning with linear function approximation for the game Snake, but I doesn't seem to get it working: the weights all eventually turn NaN and I have no idea why. Maybe ...
1
vote
0answers
12 views

Initial weights Feed Forward NN

I am trying to understand the purpose of Xavier's initialization of the weights in an ANN. I get that the main reason is that we don't like our linear combinations in the units to be very large as the ...
0
votes
0answers
111 views

How to compute derivative of loss function with respect to weights in forward neural networks

Consider the feature space $\mathcal{X}=\mathbb R^{d}$ and $\mathcal{Y}=\{1,...,c\}$ such that $c > 2$. We consider some activation function $\alpha: \mathbb R^{c} \to \mathbb R^{c}$ and out weight ...
0
votes
0answers
19 views

How to deal with the loss exploding for LSTM regression task [duplicate]

I am training a LSTM for regression problems, but the loss function randomly shoots up as in the picture below: I tried multiple things to prevent this, adjusting the learning rate, adjusting the ...
2
votes
0answers
42 views

How to Efficiently Finding All Local Maxima in a Large Parameter Space

I am working in 8-D parameter space, where every parameter is on the interval [0, 1]. The number of local maxima in this space and how they are positioned relative to one another is way more ...
3
votes
1answer
67 views

Bayesian Optimization vs. gradient descent

I don't know if this is the right place to ask this question. If you think this question is better asked in another StackExchange, please point me to that. This question is about the sampling ...
0
votes
0answers
29 views

Main idea behind reparametrization trick (distribution to function)

If I got the idea correctly, one of the main concepts behind the reparametrization trick, first presented in Kingma, D. P., & Welling, M. (2013), Auto-encoding variational bayes (ArXiv Preprint ...
2
votes
0answers
60 views

LSTM backpropagation gradient regarding vanishing and exploding gradients problem

I was looking around for a good explanation as to why LSTMs are better able to handle vanishing and exploding gradients compared to vanilla RNNs. I know it is due to the cell memory $c_t$ acting as a ...
1
vote
1answer
38 views

Constrained optimization with gradient descent

Suppose I want to maximize the likelihood $L(\theta_1, \theta_2)$ for some constraint for example $\theta_1 + \theta_2 = 1$ and no other constraints Can I just replace $\theta_2$ by $1 - \theta_1$ in ...
2
votes
0answers
39 views

Problem of jensen shannon

In GAN we want to minimize Jensen-Shannon distance and we use gradient descent. When can't we use this approach? What attribute might the training data and the distribution of the generating network ...
0
votes
0answers
19 views

Almost Perfect Accuracy in Both Training and Validation Sets, but Nothing Showed Up in All But One of the Classes' Saliency Map

My convolutional neural network (with 5 layers: first 3 are Conv2D, last 2 are FC's) to classify four different classes of protein images resulted in very high accuracies and low losses in both ...
1
vote
1answer
40 views

Gradient descent and Backpropagation

I think I understood the principles of gradient descent and backpropagation. But I think, so far, I'm not sure how they work together. Gradient descent itself is "just" an optimization ...
0
votes
1answer
23 views

When do Adaptive Optimization Algorithms modify their parameters?

When do "Ada" optimizers (e.g. Adagrad, Adam, etc...) "adapt" their parameters? Is it at the end of each mini-batch or epoch?
2
votes
1answer
67 views

Does gradient boosted trees actually use regression trees for classification, and if so, what does the gradient update?

I have often read that gradient boosting algorithms fit sequential models to the overall model's residuals, but I can't make sense of this for classification problems (for instance, what is the "...
0
votes
1answer
43 views

What is the Purpose of calculating SSE, MSE (or other metrics) if linear regression (OLS) is minimizes sum of squared errors?

Ordinary Least Squares regression is defined as minimizing the sum of squared errors. So after doing this regression (OLS) then what is the purpose of optimizing SSE (or MSE, RMSE etc.) if linear ...
1
vote
1answer
98 views

When to use weight decay for ADAM optimiser?

If you use weight decay for gradient descent (ADAM specifically) do you need to use regularisation for loss function? I believe the answer is yes since the gradient descent involves the ...
1
vote
2answers
62 views

Why do we need gradient in gradient descent?

In gradient descent algorithm, the update rule of vector parameter is as follow: From this formula, i think that the update rule only depends on the sign of the gradient. So why don't we just use ...
4
votes
2answers
76 views

Why does it appear impossible to fit Gaussians to arbitrary probability density functions $p$?

I want to fit a Gaussian $q$ to a pdf $p$ by minimizing the energy $E = -\int q(x) \log p(x) dx$. This should result in a "delta function" Gaussian with $\sigma \rightarrow 0$ and $\mu \...
1
vote
1answer
74 views

Is it normal for training loss to plateau before decreasing?

Is this training loss graph normal - where it flattens for quite a while before dropping? This is something that I am seeing when I train my neural net every time. Because whenever I read papers the ...
1
vote
1answer
33 views

Convergece of Steepest Descent

Why does Steepest Descent converge? I know that will be take the objective $f$ and walk it through direction $-\nabla f$ with step size $\alpha_k$ but step size seems able to be negative and it does ...
0
votes
0answers
6 views

What is the training process of 1 input - 2 output network based on one mixed loss?

Let's say I have a neural network with one input, 2 output layers, 1 hidden layer. I define a combined loss based on the first and second output. For simplicity, assume that the first loss is RMSE and ...
2
votes
1answer
91 views

gradient descent in neural network

Given that almost all the activation functions in neural networks are increasing, by the gradient descent rule, all parameters should be updated in the same direction (negative direction). Then how ...
0
votes
2answers
48 views

How to do gradient descent when parameter is positive definite matrix

So, suppose I have an objective function $\mathcal{L}(\Sigma)$ where $\Sigma$ is a positive definite matrix. Now, I want to optimize this function using gradient descent. Now, I think if I compute the ...
0
votes
0answers
26 views

Multiclass softprob (cross-entropy) objective gradient and Hessian formula (xgboost)

I'm trying to understand how to write a custom objective function for xgboost. I'd like to base it on the multi:softprob objective. I couldn't find the formulas for gradient and Hessian of the multi:...
1
vote
0answers
68 views

Maximum Likelihood estimation using steepest descent method

Consider the observations: $x_1=-0.24,\quad x_2=0.31,\quad x_3=2.3,\quad x_4=-1.1$ sampled from a normal distribution. I want to determine the maximum likelihood estimate of the mean $\mu$ and ...
2
votes
1answer
220 views

What exactly is the gblinear booster in XGBoost?

Aside from ordinary tree boosting, XGBoost offers DART and gblinear. On DART, there is some literature as well as an explanation in the documentation. However, I can't find any useful information ...
0
votes
1answer
12 views

What is the relationship between momentum method for gradient descent defined in machine learning and the momentum defined in physics

The momentum method for gradient descent in machine learning is defined as $$ \begin{array}{l} \mathbf{v}_{t} \leftarrow \beta \mathbf{v}_{t-1}+\mathbf{g}_{t-1} \\ \mathbf{x}_{t} \leftarrow \mathbf{x}...
0
votes
0answers
27 views

Least squares fit with polynomials: Order of the polynomial vs accuracy of the predictions

I have noisy evaluations $y_i$ of some unkown function $f$ at points $\vec{x}_i$ clustered around point $\vec{x^*}$. Now I want to fit a polynomial model to this data to get some surrogate model $\hat{...
0
votes
0answers
25 views

Gradient Boosting vs Gradient Descent

The section 10.10.2 of ESL claims that the difference between gradient boosting and steepest descent is that The tree components $t_{m} = T(x_{1};\theta_{m}),...,T(x_{N};\theta_{m})$ are not ...

1
2 3 4 5
18