2
$\begingroup$

I have a question regarding the performance of MAPE and SSE in fitting datasets generated from another equation.

I have the following equation (I am truly sorry, I do not know how to write in LaTeX): f(x) = exp(x), whose data is generated from 0.1 < x < 3.1, with a stepsize of 0.1.

What I did is I fitted the quadratic equation g(x) = Ax^2 + Bx + C using SSE and MAPE, and got totally different results, the graph shown here: enter image description here

While I know that this is expected considering that I used different objective functions in minimizing the errors, I would like to understand why is this so. As I can see from the graph, the SSE seems to be performing well on average, but with MAPE, the model is performing better than the one calculated by SSE, except at the larger values of x.

$\endgroup$
3
  • $\begingroup$ By SSE do you mean MSE? $\endgroup$ Mar 5, 2021 at 6:26
  • $\begingroup$ @user2974951: whether we minimize the sum or the mean of the squared errors makes more difference - it's just a scaling by a constant, the (reciprocal of the) number of data points. $\endgroup$ Mar 5, 2021 at 7:33
  • $\begingroup$ Should of course be "no" difference... $\endgroup$ Mar 5, 2021 at 8:39

1 Answer 1

2
$\begingroup$

I'll abstract from the misspecification (the true relationship between the predictor and the observation is exponential, but you are fitting a polynomial) and assume our models are correctly specified.

the SSE seems to be performing well on average, but with MAPE, the model is performing better than the one calculated by SSE, except at the larger values of x

Key question: what does "performing well" mean? Whether something performs well or not presupposes a metric. Whether car A performs "better" than car B depends on whether you are more interested in its fuel efficiency, or its handling, or its safety record.

Here, whether a model "performs well" can only be answered by recourse to some error metric. We have two such metrics in play here: the MAPE and the SSE. I very much assume that the model that was fitted to minimize (in-sample) MAPE performs better than the model that was fitted to minimize SSE. And vice versa.

To repeat, there is no absolute sense in which a model performs well or not - this can only be said relative to an error metric.


the SSE seems to be performing well on average, but with MAPE, the model is performing better than the one calculated by SSE, except at the larger values of x

Actually, the larger values of your predictor $x$ are a red herring. What is relevant are the larger values of the observations $y$.

Suppose we are looking at one particular observation $y_i$, and our model fit is $\hat{y}_i$. The error contribution is $(y_i-\hat{y}_i)^2$ to the SSE error metric, and $\frac{|y_i-\hat{y}_i|}{y_i}$ for the MAPE error metric. So for instance, if $y_i=1$, then a $\hat{y}_i$ that is off by $1$, or $\hat{y}_i=2$, will contribute $1$ to the SSE, and also $1$ to the MAPE. However, if the true observation is $y_i=2$, then a fit that is off by the same amount, $\hat{y}_i=3$, will still contribute $1$ to the SSE, but only $\frac{1}{2}$ to the MAPE.

The key observation here is that the percentage in the MAPE means that errors corresponding to larger actuals $y$ will get a lower weight in the error calculation. Consequently, our model fitting process will make less of an effort to fit the larger $y$ values, and rather concentrate on getting the smaller $y$ values right.

Which is what we see in your plot.


Here is a little simulation. Assume we sample some data, where the predictor is uniform on [0,1], and the outcome follows a quadratic relationship with the predictor, with some noise. Below, I'm plotting the simulated observations, together with the true relationship (the dashed black line), the fit that minimizes the SSE (the red line) and the fit that minimizes the MAPE (the green line).

Note that because the MAPE prefers under-fitting, the entire green line is below the red line. As our model is correctly specified, the red line is unbiased, i.e., if we repeat the experiment many times, it will vary around the true relationship, the dashed line. The green line will, in contrast, always be biased low. And the bias will be stronger the smaller our observations are.

simulation

R code:

true_relationship <- function(xx) xx^2-2/3*xx+2/5
ss <- 0.1
n_sample <- 100
set.seed(1)    # for reproducibility
x_sample <- runif(n_sample)
y_sample <- true_relationship(x_sample)+rnorm(n_sample,0,ss)

model_SSE <- lm(y_sample~I(x_sample^2)+x_sample)
model_MAPE <- optim(par=coefficients(model_SSE)[c(2,3,1)],
    fn=function(params)mean(abs(params[1]*x_sample^2+params[2]*x_sample+params[3]-y_sample)/y_sample))

xx <- seq(0,1,by=.01) # for plotting
opar <- par(mai=c(.5,.5,0,0))
    plot(xx,true_relationship(xx),type="n",ylim=range(y_sample),xlab="",ylab="",las=1)
    points(x_sample,y_sample,pch=19,cex=0.6,col="darkgray")
    lines(xx,true_relationship(xx),lty=2,lwd=2)
    lines(xx,predict(model_SSE,newdata=data.frame(x_sample=xx)),col="red",lwd=2)
    lines(xx,(cbind(xx^2,xx,1)%*%model_MAPE$par)[,1],col="green",lwd=2)
par(opar)
$\endgroup$
1
  • $\begingroup$ Thank you very, very much! This was what I was looking for. $\endgroup$ Mar 5, 2021 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.