0
$\begingroup$

It is my understanding that the quantile loss is not differentiable (at 0) so base gradient descent cannot be used.

However, Vowpal Wabbit which is an SGD-based learner very much includes quantile regression, and calculates the derivative as:

float e = label - prediction;
if(e == 0) return 0;
return e > 0 ? -tau : (1-tau);

Am I missing something here? Is this an accepted way to use SGD with QR?

$\endgroup$
0
$\begingroup$

Many things in applied statistical computing (and computer science in general) is based on approximations. In case of absolute loss function, it has constant decrease rate for loss < 0 and constant increase rate at loss > 0, it neither increases, nor decreases at zero. That's about theory, but in practice, it is unlikely that you will hit exact zero. Moreover, it is not differentiable at zero, but since it is neither increasing, nor decreasing, 0 is a reasonable value to use if we need to choose something for our algorithm to work. If error is equal to zero, then we are done with training and not need to update the weights any more, so this makes perfect sense.

$\endgroup$
  • $\begingroup$ So we basically ignore the mathematical properties of the function and say that this is a "good enough" approximation? $\endgroup$ – Bar Nov 7 '18 at 13:45
  • $\begingroup$ @Bar can you point any harm in setting the derivative to zero in here? It is zero when error is zero. If error is zero, then we are done with training and not need to update the weights any more, so this makes perfect sense. $\endgroup$ – Tim Nov 11 '18 at 12:15
  • $\begingroup$ I don't see any harm practically, but the derivarive is undefined for 0 mathematically. I guess we can justify it as selecting a value from the subderivatives. $\endgroup$ – Bar Nov 12 '18 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.