In [1], Schraudolph presents an algorithm for updating a weight $w$ and some auxiliary variables $p$ and $v$ given a vector gradient $g$ and another vector "$Cv$". The update rules attempt to make it so that v goes to $C^{-1}g$ where $C$ could be, for example, the Hessian. $Cv$ is a vector like $g$ that is passed into the algorithm.

The update rules are: \begin{align}\DeclareMathOperator{\diag}{diag} \DeclareMathOperator{\max}{max} w_{t+1} &= w_t - \diag(p_t)g \\ p_t &= \diag(p_{t-1})\max\left(\frac12, 1 + \mu\diag(v)g\right) \\ v_{t+1} &= \lambda v_t + \diag(p_t) (g - \lambda Cv_t) \end{align} where $\mu$ and $\lambda$ are constants.

How would you modify these update rules so that the application of training can be weighted by $0 \le u$ such that it's similar to training $u$ times with the unmodified algorithm.

My first attempt was to replace $g$ with $ug$ and replace $\lambda$ with $\lambda^u$. I'm looking for insight.

[1]: Schraudolph, N. N. (2002). Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14(7), 1723–38.

  • $\begingroup$ Is not weighting g by u is enough? Why do you modify lambda? $\endgroup$ – soufanom Feb 5 '13 at 4:06
  • $\begingroup$ @soufanom: I figured that because $λ$ is a forgetting constant, then training multiple times should forget more of $v$. $\endgroup$ – Neil G Feb 5 '13 at 4:21
  • 1
    $\begingroup$ @soufanom: You may be interested in the answer below by the author of the paper himself! $\endgroup$ – Neil G Feb 13 '13 at 6:21

Interesting idea, but after some thought I would not modify lambda.

In practice, one tries to pick a lambda as close to 1 as possible; there are two reasons why one typically ends up with a lambda < 1:

a) numerical instability of the iteration at lambda = 1;

b) the need to forget old data in nonstationary applications.

In both these contexts, the notion of weighting an input pattern as analogous to training on it a proportional number of times no longer holds up, so modifying lambda here is in my eyes overextending that analogy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.