3
$\begingroup$

Is there any reasons not to use MAE as cost function in logistic regression? Yes, I know about problems with optimization, I know that this estimation won't be max-likelihood estimation. But is there something else? Another pitfalls?

$\endgroup$
3
$\begingroup$

Here is how this could give unexpected results if you are using logistic regression for prediction. Suppose there is actually nothing to model, just a straightforward binomial distribution with $p=0.3$. We can predict anything between $0\leq\hat{p}\leq 1$. What will the expected MAE and MSE for various $\hat{p}$ look like? Let's simulate 1000 draws.

set.seed(1)
gold.standard <- runif(1e3)<0.3
summary(gold.standard)

   Mode   FALSE    TRUE 
logical     707     293

Prediction <- seq(0,1,by=0.01)
MAE <- sapply(Prediction, function(xx)mean(abs(xx-gold.standard)))
MSE <- sapply(Prediction, function(xx)mean((xx-gold.standard)^2))

plot(Prediction,MAE,ylim=range(c(MAE,MSE)),type="l",col="red",ylab="Error",lwd=2)
lines(Prediction,MSE,lwd=2)
legend("topleft",inset=.02,lwd=2,col=c("red","black"),legend=c("MAE","MSE"))

MAE vs MSE

The expected MSE is minimized, as expected, by $\hat{p}=p$. (More precisely, by $\hat{p}=0.293$ because of our simulation.) But the expected MAE is minimized by $\hat{p}=0$, i.e., by always predicting the most common outcome, FALSE. This may be what you want, but it typically isn't. (And one would usually argue that if you want a classification prediction, then you should really aim at getting an unbiased $\hat{p}$, then base your classification on whether $\hat{p}$ exceeds a threshold that balances the cost of Type I and II errors - not bias $\hat{p}$ itself.)

The key thing you should keep in mind is that minimizing the MAE will bias your predictions towards the median of their distribution (whereas minimizing the MSE will asymptotically yield unbiased predictions: you'll get their expectation). See, e.g., Hanley et al., 2001, The American Statistician.

An analogous problem plagues forecasts of low volume count data - if you assess your forecast quality using the MAE, you'll get "optimal" forecasts that may be a flat zero. I have written a little paper on this (Kolassa, 2016, IJF) which may have some helpful pointers to further literature and in fact contains an analogue to the graph above.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.