Is it acceptable to scale a model based on "intuition"?

Question

Suppose that a model has been produced that will predict the total number of customers in a business.

You are generally happy with the distribution of the model and believe the trends shown by the forecasts are fairly reasonable e.g. The shop expects more customers in the holidays and less in other months, for instance we may see more customers at Christmas and then less in January.

However the model seems to be over predicting the number of customers, maybe it says you will see 1000 customers in a month when you know that it is very unlikely you will see that. For instance in Jan 17 you had 300 customers and then 600 in Jan 18. The model is predicting Jan 19 to have over a 1000 customers yet you are aware that Jan 17/18 were large periods of growth and you are now not expecting to see such a large spike.

Would it be advised/useful to scale the model by some percentage?

So instead of predicting 1000 customers in Jan 19 we say we will only see 75% of what the model predicts. So now we have a forecast of 750 customers in Jan, which seems more reasonable.

Or is this terrible practice?

Answer 1

Based not on mathematical and statistical considerations, but on professional experience, I disagree with ERT. I work in retail demand forecasting and the procedure you just described is close to the approach that most retailers use for managing their forecasts, it is called exception based forecast management.

A retailer typically needs to manage millions of time series forecasts (tens of thousands of products × hundreds of stores) which are generated automatically. It is unrealistic to expect human analysts to be able to validate each and everyone of those.

So instead some monitoring process is setup which alerts the forecasters whenever the sales forecast passes one of several thresholds. The analyst will then step in and rescale the forecast according to their business knowledge.

The scenario you mentioned occurs, where the automatically generated forecast is possibly too large and needs to be reduced.

The reverse scenario is also possible, where the automated forecast produces a reasonable forecast, but the analyst knows that there will be some event or promotion occurring and decide to adjust the forecast upwards.

A typical real world example is the following: Based on historical sales data, my model will predict that this summer I'm going to have very high sales of Adidas shoes (they all sold out last year and the year before) - but a retail fashion forecaster with domain knowledge knows that this year the hip shoes are now Vans, not Adidas. We have yet to have reliable ML or statistical models that predict such a shift (and not for lack of trying). So they will go into the system and scale Adidas down and Vans up.

Answer 2

The models in the industry are routinely "tuned." The adjustments have different names like "overlay", "multiplier","tuning parameter" etc. These are done through either "calibration" or "expert judgement." Whatever you call it, people do this stuff all the time.

For instance, this is from Andrew Davidson's mortgage model fact sheet, "Tuning the Models" section: "We make tunings available to allow the user to customize model output to a specific portfolio’s performance as well as to allow AD&Co to issue formal tuning recommendations to the model when market conditions change." You can literally scale the output of model components using these tuning parameters.

Calibration happens when you get a vendor model, and want to apply to your problem, and can't assure that the vendor's dataset is from the same distribution as yours. For instance, the vendor model was trained on a balanced set of women and men, and you have more male customers. In this case you can calibrate the model to your sample by many different - usually crude - means.

There are ways to do this in a more systematic manner. For instance, robustification by "intercept correction." Suppose that you have a model $y=\beta_0+\beta x +\varepsilon$. The theory says that the biggest problems to the forecast $\hat y$ come from the mean shift, i.e. when $\beta_0$ moves. So, the solution is to adjust it to a new value $\tilde\beta_0$ , which is called "intercept correction".

Answer 3

No, don't do that. You should use model output to inform your decisions.

You should look into making the predictive accuracy of your model higher by doing out-of-sample testing to minimize MSEP (mean squared error of prediction) if that is your target objective.

Utilizing a model which is able to handle multiple-seasonality would be of interest to you considering this model has a retail application. Retail sales fluctuate during holidays and weekends, implying you likely will be dealing with this problem in your model.

Answer 4

This is an interesting practical case. Here is my two cents: Sometimes it is better to change your scaling. I mean instead of predicting the number of customers in your question, you can predict a percentage of maximum number of customers.

Based on practical experiments, you can define the closest number to the maximum number of customers that your company may have. and instead of predicting the exact number of customers, you can change your target value to the percentage of of customers that you have from the total. Then you can predict these percentage values for your future periods and to have exact predictions you can multiply it to your upper bound value which primarily you used to compute the percentages. In this way, you will never over estimate and you can have better predictions.