The gradient-descent tag has no wiki summary.
0
votes
0answers
10 views
Debugging Multinomial Logistic Regression Implementation [migrated]
Hi for the University we are required to implement from scratch (using numpy / scipy) Multi-logit. So basically I have to implement the log-likelihood and the graditen of that function.
I have ...
2
votes
0answers
123 views
Matrix factorization and gradient descent for recommender systems; user bias?
I've been reading about using Matrix Factorization techniques to do collaborative filtering. A popular thing to do seems to be to add user and item biases into the ratings prediction. What I don't ...
1
vote
0answers
42 views
Question about asymptotics of steepest descent method in the context of adaptive filtering
The model which will be used is defined as
$e(n) = d(n) - y(n)$
with
$y(n) = x(n)^Tw(n)$.
where $e(n)$ is the error term of the n-th observation, $x(n)$ the input vector of the n-th ...
1
vote
0answers
67 views
Batch gradient descent versus stochastic gradient descent
Suppose we have some training set $(x_{i}, y_{i})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as ...
4
votes
3answers
192 views
Why doesn't k-means give the global minimum?
I read that the k-means algorithm only converges to a local minimum and not to a global minimum. Why is this? I can logically think of how initialization could affect the final clustering and there is ...
0
votes
0answers
25 views
Help in calculating of gradient [closed]
I have a function $f(D) = \sum_{n=1}^N \frac{1}{2}(o_n-Dy_n)^T*\phi^{-1}*(o_n-Dy_n)$
I want to know what its gradient with respect to $D$ will be . I tried a couple of things but I am not sure what ...
2
votes
0answers
99 views
Kernel SVM in primal training with Stochastic Gradient Descent
In short: I am currently reading Online Learning with Kernels (http://books.nips.cc/papers/files/nips14/AA33.pdf) for fun and I can't figure out how he got to equation 8 from equations 6 and 7.
The ...
2
votes
1answer
152 views
Determine the optimum learning rate for gradient descent in linear regression
How can one determine the optimum learning rate for gradient descent? I'm thinking that I could automatically adjust it if the cost function returns a greater value than in the previous iteration (the ...
8
votes
2answers
284 views
What is the difference between EM and Gradient Ascent
I can't understand what is the difference between the algorithms EM (Expectation Maximization) and Gradient Ascent (or descent).
Is there any condition under which they are equivalent?
Thank you.
0
votes
1answer
102 views
Basic question about gradient descent
I am trying to understand how gradient descent works. My understanding is that the update rule works as follows:
$$
x_{n+1} = x_n - \alpha f'(x_n)
$$
This seems magical to me.
How did this update ...
2
votes
0answers
64 views
Random forest like techniques (bagging, random feature subset) for SGD methods
Are there any well-known results/tools/literature on using bagging and random feature subset selection for regression or SGD-based methods?
0
votes
0answers
82 views
Iterative maximum likelihood estimation of history
The model is the following:
You receive a hidden integer h from 1 to N (uniform distribution)
You take an action a numbered between 1 to M.
Describing the entire model is a MxN probability matrix ...
1
vote
2answers
101 views
Computing the likelihood gradient on a simple directed graphical model with hidden unit
SHORT VERSION:
We have a ('visible') random variable $X$ and a ('hidden') random variable $Z$. We have chosen appropriate distributions $P(X|Z)$ and $P(Z;w)$ where $w$ is the parameter of the model. ...
4
votes
1answer
170 views
Can a model of P(Y|X) be trained via stochastic gradient descent from non-i.i.d. samples of P(X) and i.i.d. samples of P(Y|X)?
When training a parameterized model (e.g. to maximize likelihood) via stochastic gradient descent on some data set, it is commonly assumed that the training samples are drawn i.i.d. from the training ...
2
votes
2answers
215 views
How to know the stochastic gradient descent is converging
How can I know if the stochastic gradient descent algorithm is converging. I cannot plot my objective function and take its average for lets say every 1000 iterations to see the trend.
My objective ...
0
votes
0answers
101 views
How to check if stochastic gradient descent is converging
I want to be able to check if the stochastic gradient descent is converging. I cannot plot the objective function, since it is itself very much. So how can I know if it is converging?
I know what the ...
0
votes
1answer
80 views
Different parameter values when using stochastic gradient descent
I am having some issues with stochastic gradient descent. Using batch gradient descent where I consider all the training sets I have certain parameter values which I know are correct.
My function is ...
1
vote
3answers
163 views
Issues with stochastic gradient descent
I am using stochastic gradient descent to learn a model. Here is the plot of the objective function for the iterations. I am trying to maximize the function value.
Taking the average of 500 ...
7
votes
1answer
401 views
How to define the termination condition for gradient descent?
Actually, I wanted to ask you how can I define the terminating condition for gradient descent.
Can I stop it based upon the number of iterations, i.e. considering parameter values for, say, 100 ...
0
votes
1answer
207 views
Issues while implementing stochastic gradient descent
Actually, I am trying to learn a model using stochastic gradient descent. I am using just a small subset of my entire data at a time to make each iteration. As far as I know, stochastic gradient ...
1
vote
1answer
204 views
Fitting logistic to small number of points in R
I'm trying to fit a logistic function to some data points. Each data "set" has 6 points that I'm trying to fit a seperate logistic function to.
Here is some sample code:
...
0
votes
1answer
167 views
Entering values into a linear regression / gradient descent algorithm
The instructor of the machine learning course I've been taking whipped this algorithm at my head without explaining how to apply it to a training set.
...
1
vote
1answer
166 views
Model function for discovering irrelevant dimensions with L1 regularization
For homework I have been given a 20-dimensional input $x \in \mathbb{R}^{20}$, many of which are suspected to be irrelevant. I tried using L1-norm Lasso regularization to uncover which dimensions ...
5
votes
1answer
357 views
Gradient descent vs lm() function in R?
I'm going through the videos in Andrew Ng's free online machine learning course at Stanford. He discusses Gradient Descent as an algorithm to solve linear regression and writing functions in Octave ...
3
votes
1answer
358 views
Solving for regression parameters in closed-form vs gradient descent
In Andrew Ng's machine learning course, he introduces linear regression and logistic regression, and shows how to fit the model parameters using gradient descent and Newton's method.
I know gradient ...
