# Demonstrating Overfitting in a Simple Model

I have been working with a finance team to help forecast revenue for some product data. Particularly when the series are short and difficult to forecast, their first response is to add a bunch of "driver" data (basically external regressors) to the model. I have given presentations a few times on overfitting and why adding 24 new series to predict a series with 12 data points is not an effective modeling strategy, but it never seems to stick. I have told them otherwise, but they seem to think that the over-fitting only happens when the drivers are "bad", so they feel I am telling them they are wrong about what the drivers are when I tell them the drivers don't help the model.

I have decided I want to try a more hands on approach to teaching them. I want to build an example in Excel (the finance team is very comfortable in Excel, but not with any programming language) where we can have a series we are trying to forecast along with some drivers. I want the data to have the following properties:

1. The drivers are all "real". In other words, the values of the target series is directly affected by the driver data.

2. The target series is easily modeled using linear regression (since we are dealing with Excel). Ideally the target series will be some linear combination of the drivers plus an error term, preferably with a coefficient of one for simplicity.

3. The target series can be modeled using an auto-regressive model. I want to make a second model using a few lags of the series to show how even though the series is a combination of drivers, we can still model that information using data from the series itself.

4. The auto-regressive model should out perform the linear regression model, at least for small amounts of data.

Some of the strategies and questions I have been considering when making this model:

1. Would having multiple drivers with the same seasonality help? This would make a more complicated relationship in the seasonal component of the driver model (since it would have two variables) than in the auto-regressive model (which would have one).

2. It seems like in order to work, the drivers need to each have their own error component (to generate the over fitting in the driver model). Is this true? It seems like that shouldn't negatively impact the auto-regressive model, but is there something I am potentially missing? Also is there a good rule of thumb for how big the error component should be relative to the "predictable" part of the series?

3. What type of patterns can I use for the driver data? A simple seasonal pattern would definitely work. Maybe an exponential decay function? Are there any other good auto-regressive processes I might consider?

4. What are some good explanations I can give for each driver? For two variable with the same seasonality, I was thinking I could have one be positive representing demand for their product and one be negative representing demand for a competitor's product. An exponential decay function could represent something like market saturation.

5. Are there any other strategies I can employ to get the result I am looking for?

• You may find this earlier answer of mine helpful, along with the paper linked there. – Stephan Kolassa Mar 31 '17 at 17:26
• @StephanKolassa Thanks, that is a good example of benefits of simpler models. I will have to think if there is a good way I can incorporate it in the context of "driver" data. – Barker Mar 31 '17 at 17:35