Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient-descent]

Gradient descent is a first-order iterative optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. For stochastic gradient descent there is also the [sgd] tag.

0
votes
0answers
5 views

Derivation of Perceptron weight update formula

I've started out studying Machine Learning and am currently reading up about how a single perceptron works. From the wikipedia page, my understanding is as follows: suppose we have an input sample $\...
1
vote
1answer
16 views

Differences between “in-bag” and “out-of-bag” empirical risks in the R package “mboost”

currently I am using the mboost R-package to estimate some additive models. When using the function gamboost(), you can control the hyper-parameters for boosting by using the option boost_control(). ...
1
vote
0answers
8 views

Can factor analysis be fit with gradient-based methods?

I know you can fit factor analysis using EM, but can you use gradient-based methods? If so, a reference would be great; otherwise, why not?
0
votes
0answers
22 views

Coordinate descent in integer programing: when does it work?

Denote $N_i=\{0,1,\dots,\bar{n}_i\}$ and define $N=N_1\times \dots \times N_I$. I want to minimize a function $f:N\rightarrow \mathbb{R}$. For the functions $f$ that interest me, it is very easy to ...
0
votes
0answers
17 views

Gradient descent does not converge to due noise in the data

For learning purposes I am trying to get a neural network to learn a fairly simple function (e.g. x => sin(x) + rand() * 0.01 but more complicated). I can ...
0
votes
1answer
13 views

Using gradient descent to train dual formulation of Kernel SVM

I've seen other posts about using gradient descent for the primal form, but not the dual form. In this book, the author discusses using (projected) gradient descent for the dual form: http://ciml....
0
votes
1answer
34 views

What does decay_steps mean in Tensorflow tf.train.exponential_decay?

I am trying to implement an exponential learning rate decay with the Adam optimizer for a LSTM. I do not want the 'staircase = true' version. The decay_steps for me feels like the number of steps that ...
0
votes
0answers
21 views

Adding regularization to an objective function when not using gradient descent

Using a simple example if I have a model: $$y = \beta_1 X_1 + \beta_2X_2 + {\rm error}$$ with cost function $${\rm Cost}= RSS + \alpha (\beta_1 + \beta_2)(\beta_1 + \beta_2)$$ If we were to use ...
2
votes
1answer
29 views

Xgboost and repeated measures

I am learning xgboost and am planning on running a tree model. My dataset includes repeated measures. In a GLMM I would include the ID to account for repeated measures and I'm curious if I should do ...
0
votes
1answer
12 views

Does using random minibatchs give more resilance against local minima, vs full batch gradient descent

It was my believe that one of the advantages of using minibatches, when training a neural network via gradient descent (be it "vanilla" or the latest flavour of AdaGrad), was an increased resiliance ...
3
votes
1answer
29 views

why too many epochs will cause overfitting?

I am reading the 《deep learning with python》. In chapter 4, about Fighting overfitting, I have two questions. why increasing epochs may cause overfitting? I know ...
0
votes
0answers
18 views

Optimization technique to find the minimum change required in features to cause change of classifier prediction

Assuming a dataset has N features and 4 (P,Q,R,S) categories. A classifier is to predict which of the 4 categories a given datapoint belongs. Say a classifier predicts that some datapoint belongs to ...
1
vote
1answer
26 views

How to understand whether Stochastic Gradient Descent has converged?

I am using SGD to solve for MSE function. My training set is around 50K, and I am monitoring the gradient at every epoch (once a pass is completed over all the training data). I played around a lot ...
2
votes
0answers
14 views

How to identify manifolds for an optimisation problem

I don't have much experience in topology, but I am interested to know if: • Given a particular problem and associated cost function, how would one deduce what kind of manifold this problem lies on. ...
0
votes
0answers
8 views

Mini Batch Gradient Descent Backpropapagation

I am a beginner to machine learning. I have derived the equations for backpropagation, and for the weight update for hidden layers, the update rule uses the output vector of the layer to multiply with ...
1
vote
0answers
20 views

gradient descent for logistic regression

I'm implementing (for learning purposes) a logistic regression model. I've followed this guide. Now, the author is taking the derivative of $l$, the cost function with respect to some $\beta_j$: $$\...
0
votes
0answers
13 views

How to choose the learning rate for stochastic gradient descent (via backtracking)?

I am trying to implement "from scratch" SGD and Mini Batch Gradient Descent in Matlab. I have to minimize a function like $f(x)= \sum f_i (X_i, y_i)$ where $(X_i, y_i)$ is a data point (features and ...
0
votes
0answers
6 views

How gradient decent training will affect if we use feature-crosses or high-order polynomials?

Considering Multivariate linear regression. We use feature scaling + mean normalization(feature transformation) on our features to keep them on the same scale. If we don't do that then our contour ...
1
vote
0answers
44 views

How cost function for simple linear regression behaves under different settings with batch gradient descent? [closed]

In the linear regression problem, using a simple linear model with 1 variable & with 2 model parameters, performing batch Gradient Descent(GD) & assuming I am using Mean Square error as my ...
1
vote
0answers
17 views

Why is Backpropogation used instead of Rosenblatt's learning Algo or gradient descent to train MLP's?

In roesnblatt's learning algo and gradient descent the output is calculated for each input and based on the error b/w the outputs calculated and desired outputs the weights are updated. Why is ...
0
votes
1answer
50 views

How are the weights of a CNN computed?

I am trying to understand the logic behind convolutional neural networks. To my understanding, the weights used are nothing more than an $w \times h$ matrix (a filter) and as with the normal neural ...
-1
votes
0answers
13 views

Tensorflow Gradients not flowing to first layer

I have a tensorflow model with 2 layers and a final operation that sums across the layers which then feeds into the optimization function. When I visualize the weights of the network it seems that the ...
-1
votes
0answers
18 views

Steps of mini-batch stochastic gradient descent in an episode [duplicate]

I am using the stochastic gradient descent (SGD) to do the optimization for the deep neural networks (DNN). I know that in one epoch I need to do multiple iterations of mini-batch SGD to make it ...
1
vote
0answers
26 views

Can you train deep recurrent neural network layer by layer?

Specifically for Gated Recurrent Unit, and say GRU is "layered" via but suppose it's only 2 layers deep for simplicity, and suppose the "total loss" = $L$ = $\sum{l_{t}} = \sum{error(y^{2}_{t})}$ ...
1
vote
0answers
40 views

Why Monte Carlo sampling is not needed for reparameterization trick?

To esitimate $\nabla_\theta \mathbb{E}_{z\sim p_\theta(z)}[f(z)]$, we have two options: REINFORCE: $\nabla_\theta \mathbb{E}_{z\sim p_\theta(z)}[f(z)] = \mathbb{E}_{z\sim p_\theta(z)}[ f(z)\nabla_\...
0
votes
0answers
35 views

How to select parameters for ADAM gradient descent

I am using ADAM for performing gradient descent. I am having difficulty in setting the learning rate, $\beta_1$ and $\beta_2$. Along with gradient descent, I am projecting the paramters on $L_1$ ball ...
1
vote
1answer
21 views

Gradient-Descent for Parameter search

I have a rather straight-forward algorithm for finding the maximum-likelihood parameter of a probability distribution using sub-sampling. I'm fairly confident this algorithm is not novel and was ...
1
vote
1answer
34 views

What implicit function used for gradient descent in numpy gradient?

TL;DR numpy.gradient calculates the gradient of an ndarray, but I am not clear as to what it is with respect to what original function. An example, (although I ...
0
votes
0answers
16 views

Training a perceptron on MNIST using whole images yielding perfect results

So I programmed a simple Perceptron algorithm to classify images from the MNIST dataset. My goal is to tell apart what image is a zero and what image is a one. First I trained my algorithm by ...
0
votes
0answers
19 views

How does gradient descent deal with regions where the function doesn't exist?

How do implementations of gradient descent typically handle functions that have regions where the function or its derivative isn't defined? Do they attempt to "edge" around the undefined region, or ...
0
votes
0answers
133 views

Gradient of the cross entropy loss function

I have been puzzled by how to calculate the derivative of the following cross entropy loss function underlying my neural network: CEloss = $\frac{-1}{N} \sum_{n=1}^{N} \sum_{k=1}^{K} t_{n,k} \log y_{...
1
vote
0answers
36 views

Linear least squares algorithms

I have stumbled across these two questions and accepted answers: (1) Do we need gradient descent to find the coefficients of a linear regression model? (2) Why use gradient descent for linear ...
2
votes
1answer
32 views

Backpropagation wrong? Doesn't it update dependent variables in hidden layer

In a multi layer perceptron or feedforward neural network, isn't backpropagation updating weights of the middle layers that are dependent variables? So for a particular hidden layer, it calculates all ...
2
votes
1answer
49 views

Gradient Descent and the No-Free-Lunch Theorem

I'm doing a presentation on the No-Free-Lunch theorem, since I've found that the best way to learn about a topic is to try and teach it...In order to get an idea what I'm talking about, I started ...
1
vote
0answers
117 views

Stochastic gradient descent vs mini-batch gradient descent

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
1
vote
2answers
56 views

Why is the second derivative required for newton's method for back-propagation?

I am troubled with why isn't the Newton's method used for backpropagation, instead, or in addition to Gradient Descent more widely. I have seen this same question, and the widely accepted answer ...
-1
votes
0answers
10 views

Logistic regression subgradients with different regularizers

Consider the following regularized logistic regression problem: $$\textbf{w}^* = \min_{\textbf{w} \in {\rm I\!R}^p} \mathcal{L}(\textbf{w}) $$ where $$ \mathcal{L}(\textbf{w}) = \frac{1}{n} \sum_{...
0
votes
0answers
13 views

Neural network backpropagation by gradient descent is better than conjugate gradient?

My understanding is that the conjugate gradient method is faster than gradient descent because it does less zig zags while descending. How come the state of the art papers I see all use gradient ...
1
vote
0answers
41 views

Possible questions on testing on hypothesis in A/B testing for a position of a data scientist (EDITED)

What are some possible machine learning and statistics questions on the subject of A/B testing for a position of a data scientist in Ad Tech industry working in retargeting? I'm interviewing for such ...
0
votes
1answer
36 views

SGD and quantile regression

It is my understanding that the quantile loss is not differentiable (at 0) so base gradient descent cannot be used. However, Vowpal Wabbit which is an SGD-based learner very much includes quantile ...
0
votes
0answers
8 views

Can adaptive learning rate method be used for dropout regularization?

if the neurons are deactivated randomly for each forward pass during an iteration, Can adaptive learning rate method for neural network such as RMSprop be used for the case of dropout regularization?
1
vote
0answers
21 views

Batch learning in digits recognition (MNIST database) [duplicate]

While working my way through M. Nielsen's "Neural networks and deep learning", I decided to try out some presumably silly things to really understand why they won't work and/or why it's not a good ...
2
votes
2answers
57 views

Why gradient descent of log probabilities in Alpha Go?

In the original AlphaGo paper, it is stated that the policy network is trained with the following gradient: I don't understand why this gradient makes sense. Why do we want to move the parameters ...
0
votes
0answers
16 views

Exploding Gradient Problem and Increasing Net Error even with Batch Normalization?

I've implemented just about everything there is such as standardizing the data set, implementing batch normalization but the problem is that the more training iterations i go through, the net error ...
1
vote
0answers
36 views

Influence of RMSE versus MSE on Convergence of Gradient Descent

I am working in TensorFlow and was wondering if I choosing an MSE loss function would cause different convergence behaviour when compared to a RMSE loss function. The square root will influence the ...
1
vote
0answers
32 views

Why the sign is *plus* in neural network [closed]

REFERENCE GITHUB GIST I wanted to implement Neural Network with Numpy in Python. Then I have two question. The first one is about the sign ...
0
votes
0answers
27 views

Calculating minimum of a function using Gradient Descent

I need to calculate the minimum of function : f(x) = (x − 3)2 , starting at x = 0 and with α = 1/3, by applying gradient descent. Could someone please help me here how to go about it? Found no clear ...
9
votes
4answers
2k views

Gradient descent optimization

I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a ...
0
votes
0answers
10 views

Gradient for deep neural net in matrix notation?

I am used to the notation for gradient that is written explicitly in the individual parameters. What you get is a sum of the derivatives w.r.t each parameter. But is there a way to write this down ...
2
votes
0answers
47 views

In what sense is momentum “optimal” for optimisation of neural networks?

This article states about the use of momentum in gradient descent for neural networks: A lower bound, courtesy of Nesterov [5], states that momentum is, in a certain very narrow and technical sense,...