Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

0
votes
0answers
9 views

How to get precise answer from stochastic gradient descent

I have a convex optimization problem in few variables and I have an unbiased estimator of the gradient without having the ability to evaluate the function itself. I want to do gradient descent but the ...
2
votes
0answers
13 views

Large Feature Values for Gradient Descent

Recently, I work on a linear regression model of my project. I have 200 samples, each of which has only one feature, to train my model. When I try to apply ...
1
vote
0answers
8 views

How can I implement this Robust PCA equation in a more efficient way?

I recently learned in class the Principle Component Analysis method aims to approximate a matrix X to a multiplication of two matrices Z*W. If X is a n x d matrix, Z is a n x k matrix and W is a k x d ...
4
votes
1answer
300 views

Is stochastic gradient descent pseudo-stochastic?

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times. ...
0
votes
0answers
15 views

word2vec gradient update clarification

I've started the Stanford NLP course cs224d online. I'm struggling to intuitively understand the mechanics behind word2vec, and how the gradient updates actually "work" in practice. The gradient in ...
0
votes
0answers
12 views

Will gradient descent prefer stronger signals?

Let's say we have a linear regression problem: $$ \mathbf{y} = \mathbf{X}_1\mathbf{\beta}_1 + \mathbf{X}_2\mathbf{\beta}_2 + \mathbf{\epsilon} $$ where $\mathbf{X}_1$ and $\mathbf{X}_2$ are sampled ...
0
votes
0answers
12 views

Gradient Descent method for Wolfe Duality Hyperplane Optimization

I am troubling myself by learning how to build an optimal hyperplane for a separable case using the $\texttt{iris}$ data in R. The function I am trying to maximize w.r.t $\alpha_{i}$ is: $$ L_D = \...
1
vote
0answers
35 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
0
votes
3answers
49 views

Weights not converging while cost function has converged in neural networks [closed]

I'm talking in an ideal scenario where a validation set isn't used. Without validation, as many epochs as possible are calculated. Training stops and finishes only when the loss function is minimized ...
1
vote
0answers
18 views

How to plot the gradient descent of a RNN model built using keras? [closed]

I'm exploring how an LSTM solves the problem of vanishing gradients. I have created a simple LSTM model on keras. I know that model.fit() returns a history object that stores model loss and accuracy ...
0
votes
1answer
23 views

Gradient boosting understanding of residual picture

I recently looking at the Gradient boosting using following blog https://medium.com/mlreview/gradient-boosting-from-scratch-1e317ae4587d I try to understand the picture but I need some help For ...
1
vote
1answer
50 views

Intuition behind computing gradient for a model

I'm trying to understand gradient computation. The basic formula I found looks like so: ...
1
vote
0answers
16 views

What is the difference between the validation loss on a regression task and the mean squared error?

The validation loss on regression task using mean squared error loss function is different from the mean squared error value directly calculated on the validation set. What is the difference between ...
3
votes
1answer
336 views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
1
vote
1answer
28 views

Feedforward networks: methods to avoid two neurons in the same layer learning the same weights and biases?

There are many questions on this site which have to do with "what happens when two neurons have the same weights/biases" and I am not asking about that. However, it is occasionally the case that a ...
1
vote
0answers
9 views

Nested CV with Online Learning

I have a time series binary classification dataset. I am implementing an online learning Logistic Regression algorithm in Sklearn and am cross validating with Sklearn's TimeSeriesSplit method. I am ...
0
votes
1answer
24 views

Loss during minibatch gradient descent

I have minibatch gradient descent code in Tensorflow for function approximation, but I am unsure when to calculate the loss. First, I create batches for x and y data. Then, I shuffle both these ...
1
vote
0answers
19 views

RMSProp Squared Gradients

In the RMSProp algorithm (And similar algorithms) that are used in Machine Learning in the subject of Adaptive Learning Rates, the squares of the gradients are used in the algorithm step. Is there ...
2
votes
1answer
25 views

Learning Rate impact on model building time

I wanted to know that does learning rate impact the model building time in case of Gradient Boosted Trees. I do understand that increasing the number of trees have an impact( more the trees, more the ...
1
vote
0answers
50 views

Effects of class imbalance on nn batch training

Say I have a binary classification task, where the positive class (1) is only 1% of the whole data set. Intuitively I can understand why this could be bad for the classifier as the model may learn ...
0
votes
1answer
48 views

Can you implement Replay Buffers for Reinforcement Learning when most experiences give zero reward?

Specifically, for a deep deterministic policy gradient, DDPG, to expedite the learning speed, it's recommended to use a Replay Buffer What if the reward is only given at a terminal state? Or, most of ...
2
votes
0answers
25 views

Learning Manifolds using Gradient Descent

I have a feedforward neural network $F(W): \mathbb R^d \rightarrow \mathbb R^k$ with $Relu$ activation, $m$ neurones per layer, $L$ layers and softmax on the output layer. $W$ denotes the weight ...
1
vote
0answers
51 views

Gradient descent expression help

In the section 2 of this paper. I didn't understand the following steps in the proof. If $x$ is channel input, $w$ channel weight vector and $\hat{w} = w/$ $ ||w||_2$ then for batchnorm ($BN$) we ...
0
votes
1answer
24 views

What is the rationale behind clipping gradients in binary neural networks?

In binarized neural networks, where activations are confined to -1/+1 values using the sign function, derivative of the sign function is estimated with a straight-through estimator. However, when I ...
0
votes
0answers
32 views

What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.?

I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & ...
1
vote
1answer
71 views

Variance of reparameterization trick and score function

For a function $\mathbf E_{z\sim q_\phi(z|x)}[f(z)]$(assuming $f$ is continuous), where $q_\phi$ is a Gaussian distribution, if we want to compute the gradient w.r.t. $\phi$, we have two way to do ...
1
vote
2answers
29 views

In linear regression, is finding the minimum of the parameters the same as gradient descent?

I'm taking a course on ML and have just began. Given a loss function, $$L = \frac{1}{N}\sum^N_{n=1}(t_n - w_0 + w_1x_n)^2$$ I am confused between the difference of using gradient descent (and maybe ...
1
vote
1answer
26 views

Specifics on weight update calculation in stochastic gradient descent

I understand the difference between batch and stochastic gradient descent as follows: let's say there are only two samples and two features. Let the function used to estimate $y^{(i)}$ be: $$h_{\theta}...
0
votes
1answer
12 views

Confusion about the derivation of the TD-Learning update rule

I am currently trying to understand the paper "Learning to Predict by the Methods of Temporal Differences" by Sutton. I am stuck with the following step: (From "Learning to Predict by the Methods of ...
0
votes
0answers
17 views

Implicit regularization in Linear models

Regarding Linear Neural Networks models with unique finite root loss function, without an explicit regularization, I am struggling to prove that in the case of overparmeterized models (i.e. $N<d$), ...
1
vote
1answer
107 views

Derivation of Perceptron weight update formula

I've started out studying Machine Learning and am currently reading up about how a single perceptron works. From the wikipedia page, my understanding is as follows: suppose we have an input sample $\...
1
vote
1answer
25 views

Differences between “in-bag” and “out-of-bag” empirical risks in the R package “mboost”

currently I am using the mboost R-package to estimate some additive models. When using the function gamboost(), you can control the hyper-parameters for boosting by using the option boost_control(). ...
1
vote
0answers
8 views

Can factor analysis be fit with gradient-based methods?

I know you can fit factor analysis using EM, but can you use gradient-based methods? If so, a reference would be great; otherwise, why not?
0
votes
0answers
25 views

Coordinate descent in integer programing: when does it work?

Denote $N_i=\{0,1,\dots,\bar{n}_i\}$ and define $N=N_1\times \dots \times N_I$. I want to minimize a function $f:N\rightarrow \mathbb{R}$. For the functions $f$ that interest me, it is very easy to ...
0
votes
0answers
24 views

Gradient descent does not converge to due noise in the data

For learning purposes I am trying to get a neural network to learn a fairly simple function (e.g. x => sin(x) + rand() * 0.01 but more complicated). I can ...
0
votes
1answer
33 views

Using gradient descent to train dual formulation of Kernel SVM

I've seen other posts about using gradient descent for the primal form, but not the dual form. In this book, the author discusses using (projected) gradient descent for the dual form: http://ciml....
1
vote
1answer
123 views

What does decay_steps mean in Tensorflow tf.train.exponential_decay?

I am trying to implement an exponential learning rate decay with the Adam optimizer for a LSTM. I do not want the 'staircase = true' version. The decay_steps for me feels like the number of steps that ...
0
votes
0answers
22 views

Adding regularization to an objective function when not using gradient descent

Using a simple example if I have a model: $$y = \beta_1 X_1 + \beta_2X_2 + {\rm error}$$ with cost function $${\rm Cost}= RSS + \alpha (\beta_1 + \beta_2)(\beta_1 + \beta_2)$$ If we were to use ...
2
votes
1answer
59 views

Xgboost and repeated measures

I am learning xgboost and am planning on running a tree model. My dataset includes repeated measures. In a GLMM I would include the ID to account for repeated measures and I'm curious if I should do ...
0
votes
1answer
15 views

Does using random minibatchs give more resilance against local minima, vs full batch gradient descent

It was my believe that one of the advantages of using minibatches, when training a neural network via gradient descent (be it "vanilla" or the latest flavour of AdaGrad), was an increased resiliance ...
3
votes
1answer
73 views

why too many epochs will cause overfitting?

I am reading the 《deep learning with python》. In chapter 4, about Fighting overfitting, I have two questions. why increasing epochs may cause overfitting? I know ...
0
votes
0answers
20 views

Optimization technique to find the minimum change required in features to cause change of classifier prediction

Assuming a dataset has N features and 4 (P,Q,R,S) categories. A classifier is to predict which of the 4 categories a given datapoint belongs. Say a classifier predicts that some datapoint belongs to ...
1
vote
1answer
33 views

How to understand whether Stochastic Gradient Descent has converged?

I am using SGD to solve for MSE function. My training set is around 50K, and I am monitoring the gradient at every epoch (once a pass is completed over all the training data). I played around a lot ...
2
votes
0answers
17 views

How to identify manifolds for an optimisation problem

I don't have much experience in topology, but I am interested to know if: • Given a particular problem and associated cost function, how would one deduce what kind of manifold this problem lies on. ...
0
votes
0answers
15 views

Mini Batch Gradient Descent Backpropapagation

I am a beginner to machine learning. I have derived the equations for backpropagation, and for the weight update for hidden layers, the update rule uses the output vector of the layer to multiply with ...
1
vote
0answers
34 views

gradient descent for logistic regression

I'm implementing (for learning purposes) a logistic regression model. I've followed this guide. Now, the author is taking the derivative of $l$, the cost function with respect to some $\beta_j$: $$\...
0
votes
0answers
35 views

How to choose the learning rate for stochastic gradient descent (via backtracking)?

I am trying to implement "from scratch" SGD and Mini Batch Gradient Descent in Matlab. I have to minimize a function like $f(x)= \sum f_i (X_i, y_i)$ where $(X_i, y_i)$ is a data point (features and ...
0
votes
0answers
7 views

How gradient decent training will affect if we use feature-crosses or high-order polynomials?

Considering Multivariate linear regression. We use feature scaling + mean normalization(feature transformation) on our features to keep them on the same scale. If we don't do that then our contour ...
1
vote
0answers
93 views

How cost function for simple linear regression behaves under different settings with batch gradient descent? [closed]

In the linear regression problem, using a simple linear model with 1 variable & with 2 model parameters, performing batch Gradient Descent(GD) & assuming I am using Mean Square error as my ...
1
vote
0answers
19 views

Why is Backpropogation used instead of Rosenblatt's learning Algo or gradient descent to train MLP's?

In roesnblatt's learning algo and gradient descent the output is calculated for each input and based on the error b/w the outputs calculated and desired outputs the weights are updated. Why is ...