7
$\begingroup$

I am working on a forecasting application in which forecast errors are measured using the symmetric mean absolute percentage error:

$$ SMAPE = \frac{1}{n} \sum\limits_{t=1}^n{\frac{|F_t - A_t|}{F_t + A_t}} $$

After creating my ML model and applying some Bayesian inference on data I have, I end up with a probability distribution of the possible actual values, i.e. the probability associated with each "guess". If it matters, that distribution is a beta-binomial distribution with a fixed number of trials, which means the possible results range from $0$ to some $N$.

Like the mean minimizes the mean squared error and the median minimizes the mean absolute error, what point estimate minimizes SMAPE? Is there any efficient algorithm for calculating it (or a sufficiently good approximation of it)?

Thank you in advance for any help!

$\endgroup$
  • $\begingroup$ I assume that you constrain $F_t>0$, right? Otherwise, if $F_t=A_t=0$, you run into $0/0$. On the other hand, if $F_t>0$ and $A_t=0$, then the fraction (the "sAPE") is 1 regardless of $F_t$, which is a common criticism of the sMAPE (see Wikipedia). $\endgroup$ – Stephan Kolassa Apr 9 '15 at 11:28
  • $\begingroup$ Since there is no 100 in the formula, "percentage" should be dropped. $\endgroup$ – Frank Harrell Apr 9 '15 at 11:45
  • $\begingroup$ @Stephan Kolassa, good question. I wouldn't mind restraining $F_t$ as positive (although intuitively, I think it should be a special case where $SMAPE = 0$). $\endgroup$ – Rui Gonçalves Apr 9 '15 at 14:28
  • $\begingroup$ @Frank Harrell you're right, I just copied the name in Wikipedia. I guess "relative" would be a better qualifier. $\endgroup$ – Rui Gonçalves Apr 9 '15 at 14:28
  • $\begingroup$ Do keep the "percentage". It's a very common measure in the forecasting world, even if the nomenclature is not entirely correct. $\endgroup$ – Stephan Kolassa Apr 9 '15 at 14:53
3
$\begingroup$

I don't think there is a closed-form solution to this question. (I'd be interested in being proven wrong.) I'd assume you will need to simulate. And hope that your predictive posterior is not misspecified too badly.


In case it is interesting, we wrote a little paper (see also this presentation) once that explained how minimizing percentage errors can lead to forecasting bias, by rolling standard six-sided dice. We also looked at various flavors of MAPE and wMAPE, but let's concentrate on the sMAPE here.

Here is a plot where we simulate "sales" by rolling $n=8$ six-sided dice $N=1,000$ times and plot the average sMAPE, together with pointwise quantiles:

fcst <- seq(1,6,by=.01)
n.sims <- 1000
n.sales <- 10
confidence <- .8
result.smape <- matrix(nrow=n.sims,ncol=length(fcst))
set.seed(2011)
for ( jj in 1:n.sims ) {
  sales <- sample(seq(1,6),size=n.sales,replace=TRUE)
  for ( ii in 1:length(fcst) ) {
    result.smape[jj,ii] <-
      2*mean(abs(sales-rep(fcst[ii],n.sales))/(sales+rep(fcst[ii],n.sales)))
  }
}

(Note that I'm using the alternative sMAPE formula which divides the denominator by 2.)

plot(sales,type="o",ylab="",xlab="",pch=21,bg="black",ylim=c(1,6),
  main=paste("Sales:",n.sales,"throws of a six-sided die"))
plot(fcst,fcst,type="n",ylab="sMAPE",xlab="Forecast",ylim=c(0.3,1.1))
polygon(c(fcst,rev(fcst)),c(
    apply(result.smape,2,quantile,probs=(1-confidence)/2),
    rev(apply(result.smape,2,quantile,probs=1-(1-confidence)/2))),
  density=10,angle=45)
lines(fcst,apply(result.smape,2,mean))
legend(x="topright",inset=.02,col="black",lwd=1,legend="sMAPE")

simulated sMAPEs

Something along these lines may help in your case. (Again, you will need to assume that your posterior predictive distribution is "correct enough" to do this kind of simulation - but you would need to assume that for any other approach, too, so this just adds a general caveat, not a specific issue.)

In this simple example of rolling standard six-sided dice, we can actually calculate and plot the expected s(M)APE as a function of the forecast:

expected.sape <- function ( fcst ) sum(abs(fcst-seq(1,6))/(seq(1,6)+fcst))/3
plot(fcst,mapply(expected.sape,fcst),type="l",xlab="Forecast",ylab="Expected sAPE")

expected sAPE

This agrees rather well with the simulation averages above. And it shows nicely that the EsAPE-minimal forecast for rolling a standard six-sided die is a biased 4, instead of the unbiased expectation of 3.5.

$\endgroup$
  • $\begingroup$ Thank you for this analysis! Indeed, through simulation (and also by intuition) I found out that there is a forecasting bias, as by estimating numbers slightly above the expected value we are inflating the denominator in SMAPE. For the same reason, that bias decreases as the order of magnitude (the log value?) of the hypotheses decrease - if you do that same analysis for an uniform distribution U(1,100) and another U(101,200), the bias for the latter is much smaller. $\endgroup$ – Rui Gonçalves Apr 9 '15 at 14:46
  • $\begingroup$ Unfortunately, calculating the expected SMAPE for each outcome is $O(n^2)$, a time complexity I cannot afford. Because of that, I was trying to know if there was an algorithm I was not aware of or if there was a way to approximate the minimizing forecast (i.e. maybe not the optimal one, but better that the mean or the median). $\endgroup$ – Rui Gonçalves Apr 9 '15 at 14:48
  • $\begingroup$ A biased forecast may minimize the sMAPE, but I wouldn't necessarily call it "better" than the mean... People usually subconsciously expect a "good" forecast to be unbiased. $\endgroup$ – Stephan Kolassa Apr 9 '15 at 14:54
  • $\begingroup$ I agree that "better" than the mean is questionable. However, in my case we summarize the accuracy of our forecasts in a single number - in that context, we wanted to do what we could to optimize our results for that specific measure :) $\endgroup$ – Rui Gonçalves Apr 9 '15 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.