4
$\begingroup$

$MAE=|y_{pred} - y_{true}|$

$\dfrac{dMAE}{dy_{pred}} = ?$

I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some APIs - e.g. Keras - however I see tensorflow doesn't allow it (although you can manually declare it tf.abs(tf.minus(y_pred,y_true))).

My question is: How is the derivative of MAE generally calculated (I only found this which uses an apparently complex approximation) and specifically how is it computed in tensorflow (when manually declaring) and keras?

$\endgroup$
5
$\begingroup$

The , as a function of $y_{\text{pred}}$, is not differentiable at $y_{\text{pred}}=y_{\text{true}}$. Elsewhere, the derivative is $\pm 1$ by a straightforward application of the chain rule:

$$\dfrac{d\text{MAE}}{dy_{\text{pred}}} = \begin{cases} +1,\quad y_{\text{pred}}>y_{\text{true}}\\ -1,\quad y_{\text{pred}}<y_{\text{true}} \end{cases}$$

The interpretation is straightforward: if you are predicting too high ($y_{\text{pred}}>y_{\text{true}}$), then increasing $y_{\text{pred}}$ yet more by one unit will increase the MAE by an equal amount of one unit, so the gradient encourages you to reduce $y_{\text{pred}}$. And vice versa if $y_{\text{pred}}<y_{\text{true}}$.

Skimming the paper you link, it seems like they approximate the MAE by a differentiable function to avoid the "kink" at $y_{\text{pred}}=y_{\text{true}}$.

As to what specifically is implemented in TensorFlow and keras, that is off-topic here. Best to consult the documentation, the source code or any specific help community.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, I'm guessing implementations are programmed to default to either +1 or -1 if y_pred=y_true, I'm sure I've read somewhere that's how the Relu is normally handled. $\endgroup$ – Miss Palmer Nov 9 '17 at 11:17
  • $\begingroup$ But doesn't this mean that the weighted increase or decrease is independent of the magnitude of the error? $\endgroup$ – Pranit Bankar Mar 10 at 4:54
  • 1
    $\begingroup$ @PranitBankar: yes indeed. If you are at any point that is different from the minimum, moving towards the minimum will reduce the MAE value - and the amount of the reduction does not depend on where you started. $\endgroup$ – Stephan Kolassa Mar 10 at 6:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.