5
$\begingroup$

First time posting. I have two columns of data, one for the model output and one for the actual data that has come in. I calculated the MAPE and got a percentage. I performed the following analysis, can you tell me if it is wrong in any way? I calculated the sample standard deviation of the absolute percentage errors, and divided it by the square root of the number of my samples to calculate the estimation of the standard deviation of the sampling distribution of the sample means. enter image description here = A

I then used a T Distribution (I only had 4 data points, hence T Dist.) to calculate the two-tailed T distribution standard deviation for a 95% confidence interval: =T.INV(97.5%,3) = B

Can I safely say that there is a 95% change that the population MAPE is within my MAPE (plus or minus) A*B?

$\endgroup$
6
$\begingroup$

Interesting question. I have been active in both academic and applied forecasting for quite a while, and I can't recall anyone discussing CIs for MAPEs ever.

I don't think your calculation is very helpful. As an example, assume that the true holdout actuals are lognormally distributed with log-mean $\mu=1$ and log-SD $\sigma=1$. Assume further that our point forecast is a fixed $\hat{y}=\exp\big(\mu+\frac{\sigma^2}{2}\big)$ (which is an expectation forecast, which is not the MAPE-minimal forecast for lognormal data).

Recall the definition of a CI: it is an algorithm that, when the entire experiment is repeated often, will contain the true parameter value with a prespecified frequency. (Note that this is different from "there is a 95% chance that any one given CI contains the parameter.")

We can run our experiment by simulation. I get the true MAPE by simulating $n=10^6$ actuals, then repeatedly ($10^5$ times) draw the $n=4$ observations you have. In each case, I calculate APEs, take their mean and SD and calculate a 95% CI as you did. Finally, I record whether this simulated CI contained the true MAPE or not.

The hit rate is only 76%, instead of 95%.

R code:

set.seed(2020)
fcst <- exp(mm)
actuals <- rlnorm(1e6,meanlog=mm,sdlog=sqrt(ss.sq))
true_MAPE <- mean(abs(fcst-actuals)/actuals)

n_reps <- 1e5
hit <- rep(NA,n_reps)
n_obs <- 4
pb <- winProgressBar(max=n_reps)
for ( ii in 1:n_reps ) {
    setWinProgressBar(pb,ii,paste(ii,"of",n_reps))
    set.seed(ii)    # for replicability
    actuals <- rlnorm(n_obs,meanlog=mm,sdlog=sqrt(ss.sq))
    APEs <- abs(fcst-actuals)/actuals
    CI <- mean(APEs)+qt(c(.025,.975),n_obs-1)*sd(APEs)/sqrt(n_obs)
    hit[ii] <- CI[1]<=true_MAPE & true_MAPE<=CI[2]
}
close(pb)
summary(hit)

Incidentally, we can change the experiment as follows: instead of a fixed point forecast, we can simulate $n=100$ iid "historical" observations, calculate the point forecast as their average (which, again, is an expectation forecast and not the MAPE-minimal one), then evaluate this point forecast on $n=4$ new observations, calculating a CI as above. The hit rate is pretty much unchanged.

You may find What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? helpful.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Any insight into what is failing in OPs approach? $\endgroup$ – Richard Hardy Sep 22 at 19:39
  • 1
    $\begingroup$ @Anom You should be able to get access back by combining your accounts: see stats.stackexchange.com/help/merging-accounts $\endgroup$ – whuber Sep 22 at 21:15
  • 1
    $\begingroup$ It's good to know it actually worked ;-). $\endgroup$ – whuber Sep 22 at 22:17
  • 1
    $\begingroup$ @RichardHardy: good question. I'd rather ask by what reasoning we would expect the CIs to work. With only 4 observations, we can't invoke the CLT. (Also in a time series context, the IID assumption is very questionable, and the Lyaponov/Lindeberg conditions may or may not be reasonably assumed to hold.) $\endgroup$ – Stephan Kolassa Sep 23 at 5:50
  • 1
    $\begingroup$ Sherlock: you may have more of a chance then, which will very much depend on your data. For instance, in the toy example above, neither the overall expectation forecast nor the MAPE-minimal forecasts yield normal MAPEs (just record the MAPEs and plot a histogram). Part of the reason definitely is the asymmetry of the lognormal assumption, so this may work better for symmetrically distributed outcomes. But even then, your data is likely not IID (enough)... $\endgroup$ – Stephan Kolassa Sep 23 at 5:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.