# Modifying stochastic meta-descent for variable weight

In [1], Schraudolph presents an algorithm for updating a weight $w$ and some auxiliary variables $p$ and $v$ given a vector gradient $g$ and another vector "$Cv$". The update rules attempt to make it so that v goes to $C^{-1}g$ where $C$ could be, for example, the Hessian. $Cv$ is a vector like $g$ that is passed into the algorithm.

The update rules are: \begin{align}\DeclareMathOperator{\diag}{diag} \DeclareMathOperator{\max}{max} w_{t+1} &= w_t - \diag(p_t)g \\ p_t &= \diag(p_{t-1})\max\left(\frac12, 1 + \mu\diag(v)g\right) \\ v_{t+1} &= \lambda v_t + \diag(p_t) (g - \lambda Cv_t) \end{align} where $\mu$ and $\lambda$ are constants.

How would you modify these update rules so that the application of training can be weighted by $0 \le u$ such that it's similar to training $u$ times with the unmodified algorithm.

My first attempt was to replace $g$ with $ug$ and replace $\lambda$ with $\lambda^u$. I'm looking for insight.

[1]: Schraudolph, N. N. (2002). Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14(7), 1723–38.

• Is not weighting g by u is enough? Why do you modify lambda? – soufanom Feb 5 '13 at 4:06
• @soufanom: I figured that because $λ$ is a forgetting constant, then training multiple times should forget more of $v$. – Neil G Feb 5 '13 at 4:21
• @soufanom: You may be interested in the answer below by the author of the paper himself! – Neil G Feb 13 '13 at 6:21