0
$\begingroup$

I have a doubt about including the date - year, month as dummy 'X' variables in linear regression in Python. The issue I encounter is when forecasting 'y' variable on a date where the 'year' is new. How do I proceed on this?

E.g. Suppose I have only Y variable - sales and X as 'date'. when X is dummified to 'year' and 'month' (from 2016 to 2018) and I am predicting for 1 year that include 2019 as well, then how do I choose the dummy coding for Year when it didn't appear earlier in the data?

My question is also to understand whether it is a good practise to dummify the date to categorical binary variable or convert it to some cubic spline

Improve this question
$\endgroup$
  • 3
    $\begingroup$ Dummy-coding the date means you are unwilling to use past information to model future data, making it impossible to forecast. $\endgroup$ – whuber♦ Sep 9 '19 at 21:25
  • 1
    $\begingroup$ See what type of data are dates. $\endgroup$ – kjetil b halvorsen♦ Sep 9 '19 at 21:58
2
$\begingroup$

Don't use the date or the year as a dummy variable. Don't, don't, don't.

Dummy coding is used for categorical data, e.g., car brands or hair colors. Dates and years aren't. They are interval scaled. Interval scaled data should be translated into a single predictor that counts the number of days, years (or seconds) since an arbitrary origin. (The choice of origin will influence your intercept parameter estimate.)

In forecasting out, it is often good practice to not extrapolate this predictor linearly, but to dampen it. This has the effect of dampening any trend your model fits.

Even better, take a look at a standard forecasting textbook, like this one.

Improve this answer
$\endgroup$
  • $\begingroup$ Thank you. Do you know any implementation examples? $\endgroup$ – user11812781 Sep 10 '19 at 2:50
  • $\begingroup$ Sorry, no, I don't speak Python. It should be a pretty straightforward use case, and findable in some Data Science teaching materials. $\endgroup$ – Stephan Kolassa Sep 10 '19 at 6:44
  • $\begingroup$ This is generally true, but there are some cases where treating date/year as dummies should be okay. Suppose you are interested in some relationship that you have measured on 2 or 3 irregularly spaced dates or years (not ideal, but unfortunately common in my field). You aim to draw inferences about the relationship but are not interested in the time aspect or in forecasting. You need to account for the different sampling dates, though. I imagine using dummies for date should work fine in this case. $\endgroup$ – mkt - Reinstate Monica Sep 15 '19 at 6:14
  • 1
    $\begingroup$ @mkt: I do see your point. With only two or three different dates in the training sample, I would be careful about using trends (or seasonality) in forecasting, anyway. $\endgroup$ – Stephan Kolassa Sep 15 '19 at 6:31
  • $\begingroup$ @StephanKolassa I posted a follow up question in R here If you have any suggestions on that, would be appreciated. @mkt Could you also check the question $\endgroup$ – user11812781 Sep 16 '19 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.