6
$\begingroup$

EDIT: I just want to note that both David and Richard's answers (and comments) I think are needed to paint the full picture in response. To me I still haven't felt a real need to try out time-series frameworks from the given replies if all I can about is strong predictions and have a good sense of how to set up the regression formula, but I do take the points taken on interpretation of the coefficients and overall efficiency (and that, perhaps, I'm doing it the "hard" way).


I'm a bit naive to time-series related models like ARIMA as I can't seem to find a justification for them compared to a well-setup regression model for forecasting. Numerous responses online point to the vulnerability of linear regression due to thinks like autocorrelated errors, seasonality, and extrapolation, but it seems to me I can accommodate much of this with good data prep:

  • Seasonality - Model it. If there are periodic dips, I've had good luck catching them with various flag columns (month, quarter, even years, etc).
  • Autocorrelation - Include lagged values. Taking deltas, rolling means, and various statistics against prior values seems to help nicely.
  • Extrapolation - I don't see how time-series approaches navigate this any better since rare to no series is truly stationary into the future. I've found that keeping a running count from an initial reference point (i.e. months since start) seems to help general directional trends, and at some point you need to retrain anyways.

Adding on to that, regression models allow for a standard means of including multiple other input features (for instance market data if predicting sales) that do help that extrapolation quandary, as well as the benefits of a typical "optimize it to heck" framework of machine learning... I've never found the justification for a true time-series route.

Can someone help explain what I'm missing in practical use case terms? Or is it perhaps just that regression models do require that careful data modelling before use?

$\endgroup$
6
  • $\begingroup$ stats.stackexchange.com/questions/35510 contains a discussion of the spatial version of this question. $\endgroup$
    – whuber
    Jun 25 at 21:32
  • $\begingroup$ Regarding your initial note (due to the edit): my answer tells explicitly that time series approaches will be superior in forecasting in a nontrivial subset of cases. Given that, I wonder if we are on the same page. $\endgroup$ Jun 29 at 13:55
  • $\begingroup$ @RichardHardy Perhaps. My general take-home is that: 1) For less "work" (I might quibble on this due to needing to tune hyperparameters), I would expect better predictions via a time-series approach like ARIMA or ARIMAX if including other predictors, 2) If I want to look at causation, time-series is likely needed to get meaningful p-values and coefficients, 3) However, a carefully tuned/setup regression model like in the original post might catch up to time-series method's predictive power or (possibly) surpass it in some situations. $\endgroup$
    – Josh
    Jun 29 at 16:09
  • $\begingroup$ I am fine with 1 and 2. Regarding 3, surely there will be some situation where regression can catch up. But what you wrote in the edit seemed to be a bit more than that (and too strong in my opinion): I still haven't felt a real need to try out time-series frameworks <...> if all I can about is strong predictions. $\endgroup$ Jul 1 at 11:02
  • $\begingroup$ @RichardHardy That's fair and likely more speaks to my personal motivation and time to learn to new things than anything else. For me, when I can get "pretty decent" results using a very flexible and familiar paradigm, it's hard to allocate time for a new paradigm of perhaps unclear superiority when busy. What I would be very interested in is some efficacy comparison research between the methods in fair comparison conditions (i.e. both getting retrained the same amount, same time steps, etc), but that's perhaps another deeper question. $\endgroup$
    – Josh
    Jul 1 at 20:27
7
$\begingroup$

It is somewhat of a false dichotomy that one has choose between using a time series framework OR a regression framework.

The main reason for wanting to use a time-series framework would be autocorrelated errors. With this in mind, one can incorporate autocorrelated errors into regression by using Regression with ARMA errors (The ARIMAX model muddle - Hyndman) and in a sense get the 'best of both worlds'.

$\endgroup$
8
  • $\begingroup$ I may need some help understanding the autocorrelated errors bit. While I know what they are in theory, I have a hard time seeing how they bother me in regression in practice once set up with things like lags and control variables as framed in the question. $\endgroup$
    – Josh
    Jun 25 at 23:01
  • $\begingroup$ @Josh, see "Correcting for autocorrelation in simple linear regressions in R" for an answer. Here are some other variations on the theme. $\endgroup$ Jun 26 at 7:48
  • 1
    $\begingroup$ @Josh the biggest advantage is interpretability. See the section ARMAX models in the Hyndman blog post I linked. Basically your $\beta$ values become hard to interpret if you also include lagged values of your response variable in your regression equation. However if you use Regression with ARMA errors your regression $\beta$'s 'make sense' in the same way as they would if there was no autocorrelation. $\endgroup$ Jun 26 at 12:42
  • $\begingroup$ @David That makes sense I think, though I would need to study the implementation. Ignoring interpretability, would I expect more predictive power, or it ends up roughly (possibly) the same in my proposition? $\endgroup$
    – Josh
    Jun 26 at 14:29
  • $\begingroup$ @Josh it is hard to say exactly which method has more predictive power. What I would expect is using regression with ARMA errors you will probably end up with more accurate estimates of your $\beta$ coefficients which will help with prediction. $\endgroup$ Jun 26 at 19:00
4
$\begingroup$

Autocorrelation
Here is a counterexample. Suppose the data generating process is either exactly an invertible MA($1$) model or well approximated by one. If you wanted to approximate it by a time series regression about as well, you would need an AR($\infty$) model. This is not feasible, so you would end up with an AR($p$) with some large $p$. Estimating the $p+2$ parameters* of the AR($p$) model will make it have high variance and thus perform poorly in forecasting. Meanwhile, you could use an MA($1$) model instead. It has only 1+2 parameters* and thus much lower variance and better forecast accuracy.

Seasonality
Here is another counterexample. Consider a parsimonious SARIMA model as a data generating process or its close approximation. The seasonality generated by a SARIMA model cannot be modelled by dummy variables (flags as you called them). Moreover, approximating it with a time series regression might take a lot of variables and bring in a lot of unnecessary estimation variance.
Even in the very simplest instance of SARIMA(1,0,0)(1,0,0), the SARIMA model will have 2+2 parameters* to be estimated while an equivalent autoregression (which you can consider as a time series regression) will have 3+2 parameters. This way you would be estimating one superfluous parameter and thereby increasing the variance of the model. If we were to add some moving average terms, this would get significantly worse.

Extrapolation
If by that you mean forecasting into the future (beyond the available data), then the above two points apply. Otherwise I agree that there are challenges to both approaches.

Regarding including additional variables, regression with ARMA errors (as already suggested by David Veitch) is always an option. But these additional variables typically also need to be forecast, which brings us to vector autoregressions (VAR) and VARMA models.

In conclusion, it is good to know both time series regression and ARIMA-type time series models. There will be situations where the former is more natural or more effective, and there will be situations where the opposite is the case. You can then only benefit from having both tools under your belt and being able to use whichever one works better at the given task.

*+2 comes from the intercept and the error variance.

$\endgroup$
3
  • $\begingroup$ this is a bit hard to parse being written mostly from the "very familiar with ARIMA" perspective which unfortunately I noted I wasn't. It seems though the general motif of your points is that, to accomplish the same feats as time-series frameworks using multivariate regression, you 1) will generally end up with a more complicated model because after all you had to 2) try and puzzle put those variables/lags/seasonality regressors yourself. Then you get the typical problems that come with more complicated models. Is that a fair take? $\endgroup$
    – Josh
    Jun 26 at 14:39
  • $\begingroup$ @Josh, hm, sort of. I think I am mostly appealing to the no free lunch theorem of statistical learning / machine learning. It says that there is no single model/method/algorithm that works best for all problems. ARIMA and other classical time series models may be best suited for some data generating processes while regression for other. The model needs to be tailored to the problem, and ARIMA and co are well suited for some problems. Trying to conquer these problems with regression may be possible but less efficient (and less natural when it comes to interpretation). $\endgroup$ Jun 26 at 15:58
  • $\begingroup$ @josh: I don't have a general answer, but, for certain specific problems, it can be clear cut that time series approaches will be better. For example, suppose you had a problem where the response was dependent on what happened ( say the error ) in the immediately previous period or previous 2 or previous n periods. A standard regression model can't handle that case. So there are definitely cases where a regression framework is inferior. For specific examples of what I'm referring to, google for dynamic models in econometrics or lagged dependent models or ARDL models. $\endgroup$
    – mlofton
    Jul 2 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.