I am working on a problem with large and sparse data set. The logistic regression predictions using VW are satisfactory. Now, to improve the performance of the system I need to be able to tell a confidence score or confidence interval for the probability prediction. That is, even if two data samples have same prediction, the confidence score or interval should be able to differentiate the predictions that are coming from well-learned features vs predictions coming from rare features. (Consider that the variables are categorical, hence in the VW setting, each variable-value pair is treated as a feature.)
So, a few questions:
As this is an online learning setup, is there a way to calculate the standard errors of the coefficients of the features?
Can we use the
ssgrad(sum of squared gradients in adaptive updates) from the VW output as equivalent alternative to the uncertainty score proposed in this paper? Do any of the implementations like invariant updates and approximations to speed up the learning affect the purpose of using
ssgradas an uncertainty score?
Any suggestions, discussions and comments are welcome.