# Which one to compromise between MAPE and Adj R square in multiple regression

I'm trying to forecast sales of a product based on the other variables like Competitor sales, Fuel Price and CPI (Consumer Price Index).

The below given output (based on 1 to 44 months) gives me the lowest MAPE 11.62 when I validated with 45 to 48 actual sales

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)                 -2320.6320   496.3898  -4.675 3.83e-05 ***
Sales lag_1                     0.2124     0.1119   1.898 0.065515 .
Competi_sales(1) lag1          -1.6535     0.8875  -1.863 0.070404 .
Competi_Sales(1)_lag3          -5.4108     0.8352  -6.478 1.42e-07 ***
Competi sales(2)_lag1           2.3004     0.5726   4.017 0.000277 ***
Fuel price                     -48.3714    17.5225  -2.761 0.008926 **
CPI                            22.2696     3.4485   6.458 1.51e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 212.7 on 37 degrees of freedom
Multiple R-squared:  0.7252,    Adjusted R-squared:  0.6806
F-statistic: 16.27 on 6 and 37 DF,  p-value: 4.58e-09


I understand that by removing Sales lag_1 and Competi_sales(1) lag1 from the model (since both are not significant at alpha 0.05), the Adjusted R squared can be improved from 0.6806 but when If I do that the MAPE is increasing. For business use, MAPE is often preferred because apparently managers understand percentages better than other accuracy parameters.

Should I go ahead and forecast the sales using this model or should I remove the insignificant variables?

• In-sample performance may be a poor indicator of out-of-sample forecasting performance. Instead you could estimate your candidate models on a training sample, validate them on a validation sample and then select the best one (in terms of MAPE if you like) to be used for actual forecasting. Feb 11, 2015 at 7:26
• The above given Regression output is based on 44 (1 to 44) months then I validated the model with 4 months (45 to 48) which I retained for validation. I have given this information in my recent edit. I missed to mention, sorry about that.. Feb 11, 2015 at 8:58
• Only 4 obs. in the validation set is quite little... I understand your whole sample is small (48 obs.) but validation set should preferably be larger (e.g. 32 obs. for training and 16 for validation). And if you are in a time series setting, you could use a rolling window. Feb 11, 2015 at 9:13
• Thanks, Going forward I will get enough sample. Unfortunately, now I would need to predict the next 3 or 4 months sales figures with available data. Will this regression approach still work for me? Feb 11, 2015 at 9:25
• Not quite sure what you mean by this regression approach. The suggestions by @StephanKolassa make sense to me, you could very well try them out. Feb 11, 2015 at 9:31

In addition, it is not automatically the case that removing insignificant predictors will improve your adjusted $R^2$. This may happen or not.
Finally, you are including lagged sales. You may want to look at ARIMA models (e.g., auto.arima() in the forecast package, where you can include additional eXplanatory or eXternal variables like laggged competitor sales - note: which you will need to forecast again out-of-sample - or CPI via the xreg parameter.
• If you want to predict, $R^2$ is irrelevant - it's a measure of model fit. Do what the holdout sample suggests. However, remember that real data is noisy, so if the improvement in MAPE is trivial, it may turn out afterwards that the other model may have been better anyway. Feb 11, 2015 at 10:47