9
$\begingroup$

$MAE=|y_{pred} - y_{true}|$

$\dfrac{dMAE}{dy_{pred}} = ?$

I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some APIs - e.g. Keras - however I see tensorflow doesn't allow it (although you can manually declare it tf.abs(tf.minus(y_pred,y_true))).

My question is: How is the derivative of MAE generally calculated (I only found this which uses an apparently complex approximation) and specifically how is it computed in tensorflow (when manually declaring) and keras?

$\endgroup$

2 Answers 2

14
$\begingroup$

The , as a function of $y_{\text{pred}}$, is not differentiable at $y_{\text{pred}}=y_{\text{true}}$. Elsewhere, the derivative is $\pm 1$ by a straightforward application of the chain rule:

$$\dfrac{d\text{MAE}}{dy_{\text{pred}}} = \begin{cases} +1,\quad y_{\text{pred}}>y_{\text{true}}\\ -1,\quad y_{\text{pred}}<y_{\text{true}} \end{cases}$$

The interpretation is straightforward: if you are predicting too high ($y_{\text{pred}}>y_{\text{true}}$), then increasing $y_{\text{pred}}$ yet more by one unit will increase the MAE by an equal amount of one unit, so the gradient encourages you to reduce $y_{\text{pred}}$. And vice versa if $y_{\text{pred}}<y_{\text{true}}$.

Skimming the paper you link, it seems like they approximate the MAE by a differentiable function to avoid the "kink" at $y_{\text{pred}}=y_{\text{true}}$.

As to what specifically is implemented in TensorFlow and keras, that is off-topic here. Best to consult the documentation, the source code or any specific help community.

$\endgroup$
3
  • $\begingroup$ Thanks, I'm guessing implementations are programmed to default to either +1 or -1 if y_pred=y_true, I'm sure I've read somewhere that's how the Relu is normally handled. $\endgroup$
    – tea_pea
    Nov 9, 2017 at 11:17
  • 1
    $\begingroup$ But doesn't this mean that the weighted increase or decrease is independent of the magnitude of the error? $\endgroup$ Mar 10, 2020 at 4:54
  • 2
    $\begingroup$ @PranitBankar: yes indeed. If you are at any point that is different from the minimum, moving towards the minimum will reduce the MAE value - and the amount of the reduction does not depend on where you started. $\endgroup$ Mar 10, 2020 at 6:08
1
$\begingroup$

As for Tensorflow's Keras here is the good answer: https://stackoverflow.com/a/35344963

And if we will dig into the source code we will see that it uses math_ops.sign wich is return 0 for 0 https://github.com/tensorflow/tensorflow/blob/c91e944d626b517781af6a63c0aee302ab2457e3/tensorflow/python/ops/math_grad.py#L577

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.