# Tagged Questions

The tag has no wiki summary.

60 views

### In stochastic gradient descent, is there only one update to $\theta$ for each iteration?

I have read that the update equation for stochastic gradient descent is as shown below, for each iteration, k. Does one iteration correspond to one training example? So for each example is there only ...
28 views

### Issues with calculating gradient descent operation

I have this issue when using gradient ascent. I have some synthetic data and after my first iteration the objective function decreases and from the second iteration it keeps on increasing. Is it ...
24 views

### Confusion related to calculation of partial derivative

I have this function $P = f(\alpha)$. $\alpha$ is a function $\alpha = f(\theta, x)=\theta x$. Now I have $P = \alpha(x_1+x_2 + ...x_n)$ Now I need to calculate the partial derivative of P wrt ...
78 views

### What is the difference between online and batch Learning?

I currently read the paper Efficient Online and Batch Learning using Forward-Backward Splitting by John Duchi and Yoram Singer. I am very confused about the usage of the terms 'Online' and 'Batch'. I ...
140 views

### Back-propagation in Neural Nets with >2 hidden layers

We have the following update formulas: Output layer (indexed by $k \in \{1, \dots, \text{number of classes} \}$). $o_k$ - value of output neuron $k$ $d_k$ - desired value of output neuron $k$ ...
99 views

73 views

### Random forest like techniques (bagging, random feature subset) for SGD methods

Are there any well-known results/tools/literature on using bagging and random feature subset selection for regression or SGD-based methods?
91 views

### Iterative maximum likelihood estimation of history

The model is the following: You receive a hidden integer h from 1 to N (uniform distribution) You take an action a numbered between 1 to M. Describing the entire model is a MxN probability matrix ...
111 views

### Computing the likelihood gradient on a simple directed graphical model with hidden unit

SHORT VERSION: We have a ('visible') random variable $X$ and a ('hidden') random variable $Z$. We have chosen appropriate distributions $P(X|Z)$ and $P(Z;w)$ where $w$ is the parameter of the model. ...
248 views

### Can a model of P(Y|X) be trained via stochastic gradient descent from non-i.i.d. samples of P(X) and i.i.d. samples of P(Y|X)?

When training a parameterized model (e.g. to maximize likelihood) via stochastic gradient descent on some data set, it is commonly assumed that the training samples are drawn i.i.d. from the training ...
362 views

### How to know the stochastic gradient descent is converging

How can I know if the stochastic gradient descent algorithm is converging. I cannot plot my objective function and take its average for lets say every 1000 iterations to see the trend. My objective ...
112 views

### Different parameter values when using stochastic gradient descent

I am having some issues with stochastic gradient descent. Using batch gradient descent where I consider all the training sets I have certain parameter values which I know are correct. My function is ...
247 views

### Issues with stochastic gradient descent

I am using stochastic gradient descent to learn a model. Here is the plot of the objective function for the iterations. I am trying to maximize the function value. Taking the average of 500 ...
685 views

### How to define the termination condition for gradient descent?

Actually, I wanted to ask you how can I define the terminating condition for gradient descent. Can I stop it based upon the number of iterations, i.e. considering parameter values for, say, 100 ...
315 views

### Issues while implementing stochastic gradient descent

Actually, I am trying to learn a model using stochastic gradient descent. I am using just a small subset of my entire data at a time to make each iteration. As far as I know, stochastic gradient ...
417 views

### Fitting logistic to small number of points in R

I'm trying to fit a logistic function to some data points. Each data "set" has 6 points that I'm trying to fit a seperate logistic function to. Here is some sample code: ...
200 views

### Entering values into a linear regression / gradient descent algorithm

The instructor of the machine learning course I've been taking whipped this algorithm at my head without explaining how to apply it to a training set. ...
168 views

### Model function for discovering irrelevant dimensions with L1 regularization

For homework I have been given a 20-dimensional input $x \in \mathbb{R}^{20}$, many of which are suspected to be irrelevant. I tried using L1-norm Lasso regularization to uncover which dimensions ...