I am implementing a simple online logistic regression classification. For the training I use stochastic gradient descent, which is quite straight forward. The problem I was then facing is on how to normalize the input data. I have found this post: Regularization and feature scaling in online learning?, and implemented algorithm 2 of this paper: http://arxiv.org/pdf/1305.6646v1.pdf, called NAG. In the second column of page 3, the authors state that in practice they used a variation of this algorithm called sNAG, which is more resistant against outliers, and rapidly explain how to implement it:

we simply keep the accumulator $s_i = \sum x_i^2$ and use $\sqrt{s_i / t}$ in the update rule.

This makes sense as the standard deviation may be less prone to change when facing outliers.

If I understand correctly, $\sum x_i^2$ will replace line (a) ii.

What I do not understand is what the authors mean by 'in the update rule'. Do they mean replace $s_i$ by $\sqrt{s_i / t}$ in line (a) i, (c) and (d) ii of algorithm 2, or just in (d) ii, or is it something completely different?

I've been trying to access their technical report cited at the end of the paper (Ross, S., Mineiro, P., and Langford, J.(2012). Vowpal wabbit implementation of nag and snag. Technical report, github.com.), but haven't been able to find it.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.