4
$\begingroup$

I'm trying to forecast sales of a product based on the other variables like Competitor sales, Fuel Price and CPI (Consumer Price Index).

The below given output (based on 1 to 44 months) gives me the lowest MAPE 11.62 when I validated with 45 to 48 actual sales

Coefficients:                               
                              Estimate Std. Error t value Pr(>|t|)                                  
(Intercept)                 -2320.6320   496.3898  -4.675 3.83e-05 ***                              
Sales lag_1                     0.2124     0.1119   1.898 0.065515 .                                
Competi_sales(1) lag1          -1.6535     0.8875  -1.863 0.070404 .                                
Competi_Sales(1)_lag3          -5.4108     0.8352  -6.478 1.42e-07 ***                              
Competi sales(2)_lag1           2.3004     0.5726   4.017 0.000277 ***                              
Fuel price                     -48.3714    17.5225  -2.761 0.008926 **                              
CPI                            22.2696     3.4485   6.458 1.51e-07 ***                              
---                             
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1                              

Residual standard error: 212.7 on 37 degrees of freedom                             
Multiple R-squared:  0.7252,    Adjusted R-squared:  0.6806                                 
F-statistic: 16.27 on 6 and 37 DF,  p-value: 4.58e-09

I understand that by removing Sales lag_1 and Competi_sales(1) lag1 from the model (since both are not significant at alpha 0.05), the Adjusted R squared can be improved from 0.6806 but when If I do that the MAPE is increasing. For business use, MAPE is often preferred because apparently managers understand percentages better than other accuracy parameters.

Should I go ahead and forecast the sales using this model or should I remove the insignificant variables?

| cite | improve this question | |
$\endgroup$
  • 2
    $\begingroup$ In-sample performance may be a poor indicator of out-of-sample forecasting performance. Instead you could estimate your candidate models on a training sample, validate them on a validation sample and then select the best one (in terms of MAPE if you like) to be used for actual forecasting. $\endgroup$ – Richard Hardy Feb 11 '15 at 7:26
  • $\begingroup$ The above given Regression output is based on 44 (1 to 44) months then I validated the model with 4 months (45 to 48) which I retained for validation. I have given this information in my recent edit. I missed to mention, sorry about that.. $\endgroup$ – Learner Feb 11 '15 at 8:58
  • 2
    $\begingroup$ Only 4 obs. in the validation set is quite little... I understand your whole sample is small (48 obs.) but validation set should preferably be larger (e.g. 32 obs. for training and 16 for validation). And if you are in a time series setting, you could use a rolling window. $\endgroup$ – Richard Hardy Feb 11 '15 at 9:13
  • $\begingroup$ Thanks, Going forward I will get enough sample. Unfortunately, now I would need to predict the next 3 or 4 months sales figures with available data. Will this regression approach still work for me? $\endgroup$ – Learner Feb 11 '15 at 9:25
  • $\begingroup$ Not quite sure what you mean by this regression approach. The suggestions by @StephanKolassa make sense to me, you could very well try them out. $\endgroup$ – Richard Hardy Feb 11 '15 at 9:31
4
$\begingroup$

+1 to @RichardHardy's comment. In-sample fit is not a good guide to out-of-sample forecast accuracy. Relying on in-sample fits can/will lead to overfitting and poor out-of-sample performance. Instead, use a holdout sample and check accuracy on that.

I heartily recommend this free open source online forecasting textbook, especially the section on evaluating accuracy.

In addition, it is not automatically the case that removing insignificant predictors will improve your adjusted $R^2$. This may happen or not.

Finally, you are including lagged sales. You may want to look at ARIMA models (e.g., auto.arima() in the forecast package, where you can include additional eXplanatory or eXternal variables like laggged competitor sales - note: which you will need to forecast again out-of-sample - or CPI via the xreg parameter.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Provided links are very useful. This validation (MAPE) is based on hold out sample (my recent edit have this information, sorry for not mentioning this earlier). I agree with you on improving the Adj R-sqrd, if it improves by removing insignificant predictor at the same time if increases MAPE. (Let's ignore lagged predictors from this scenario) For example we have CPI, Fuel Price and Product price are the predictors and one of them is insignificant and removing that improves the Adj-R sqrd and reduces the accuracy. What's your suggestion in this scenario? $\endgroup$ – Learner Feb 11 '15 at 10:01
  • 2
    $\begingroup$ If you want to predict, $R^2$ is irrelevant - it's a measure of model fit. Do what the holdout sample suggests. However, remember that real data is noisy, so if the improvement in MAPE is trivial, it may turn out afterwards that the other model may have been better anyway. $\endgroup$ – Stephan Kolassa Feb 11 '15 at 10:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.