I created a hierarchical GAM to model final event sales as a function of sales to date, days until event, teams that are playing, and event month with a grouping at the home team level. This yielded an r-squared of 83%, a good fitted value vs residuals plot, good qq plot, etc. But when I backtest against historical data the model undershoots actual event predictions by about 30%. I previously built a very similar model for the same use case that performed very well +- 5% vs historical data.

The difference with this is that I am working with a dataset about 5% of the size of my previous one, but what I think is more important is that I am using the past 3 years of data as my training set, and the CAGR in the dataset in terms of the continuous variables like sales to date and final event sales is +75% per year (working with sports seasons, so season to season), and the number of transactions is up something like 50% annually. What I think is happening is essentially the proportional relationships between independent and dependent variables are breaking down due to the dramatic yearly growth of the dataset. Whereas in the training dataset small independent variable numbers yielded small dependent variable numbers, now small numbers are yielding large numbers holding variables like days to event and team constant.

Is there any sort of transformation I can do to the data to salvage this model? Are there other types of models that might perform better or anything I can do to this model to account for changing relationships in the data over time?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.