# SGD and quantile regression

It is my understanding that the quantile loss is not differentiable (at 0) so base gradient descent cannot be used.

However, Vowpal Wabbit which is an SGD-based learner very much includes quantile regression, and calculates the derivative as:

float e = label - prediction;
if(e == 0) return 0;
return e > 0 ? -tau : (1-tau);

Am I missing something here? Is this an accepted way to use SGD with QR?

## 1 Answer

Many things in applied statistical computing (and computer science in general) is based on approximations. In case of absolute loss function, it has constant decrease rate for loss < 0 and constant increase rate at loss > 0, it neither increases, nor decreases at zero. That's about theory, but in practice, it is unlikely that you will hit exact zero. Moreover, it is not differentiable at zero, but since it is neither increasing, nor decreasing, 0 is a reasonable value to use if we need to choose something for our algorithm to work. If error is equal to zero, then we are done with training and not need to update the weights any more, so this makes perfect sense.

• So we basically ignore the mathematical properties of the function and say that this is a "good enough" approximation?
– Bar
Nov 7, 2018 at 13:45
• @Bar can you point any harm in setting the derivative to zero in here? It is zero when error is zero. If error is zero, then we are done with training and not need to update the weights any more, so this makes perfect sense.
– Tim
Nov 11, 2018 at 12:15
• I don't see any harm practically, but the derivarive is undefined for 0 mathematically. I guess we can justify it as selecting a value from the subderivatives.
– Bar
Nov 12, 2018 at 10:02