What are relation and difference between time series and regression?

For models and assumptions, is it correct that the regression models assume independence between the output variables for different values of the input variable, while the time series model doesn't? What are some other differences?

For methods, from a website by Darlington

There are a number of approaches to time series analysis, but the two best known are the regression method and the Box-Jenkins (1976) or ARIMA (AutoRegressive Integrated Moving Average) method. This document introduces the regression method. I consider the regression method far superior to ARIMA for three major reasons

I don't quite understand what the "regression method" for time series is on the website, and how it is different from the Box-Jenkins or ARIMA method. I appreciate if someone can give some insights on those questions.

Thanks and regards!

  • 2
    $\begingroup$ Most answers and comments here focus on the more specific question towards the end. This is just a flag that time series analysis is about much, much more than Box-Jenkins or ARIMA. Entire fields of time series analysis have quite different (or at least more general) focus. Unobserved components models are just one of several examples. $\endgroup$ – Nick Cox Jul 18 '13 at 16:07

I really think this is a good question and deserves an answer. The link provided is written by a psychologist who is claiming that some home-brew method is a better way of doing time series analysis than Box-Jenkins. I hope that my attempt at an answer will encourage others, who are more knowledgeable about time series, to contribute.

From his introduction, it looks like Darlington is championing the approach of just fitting an AR model by least-squares. That is, if you want to fit the model $$z_t = \alpha_1 z_{t-1} + \cdots + \alpha_k z_{t-k} + \varepsilon_t$$ to the time series $z_t$, you can just regress the series $z_t$ on the series with lag $1$, lag $2$, and so on up to lag $k$, using an ordinary multiple regression. This is certainly allowed; in R, it's even an option in the ar function. I tested it out, and it tends to give similar answers to the default method for fitting an AR model in R.

He also advocates regressing $z_t$ on things like $t$ or powers of $t$ to find trends. Again, this is absolutely fine. Lots of time series books discuss this, for example Shumway-Stoffer and Cowpertwait-Metcalfe. Typically, a time series analysis might proceed along the following lines: you find a trend, remove it, then fit a model to the residuals.

But it seems like he is also advocating over-fitting and then using the reduction in the mean-squared error between the fitted series and the data as evidence that his method is better. For example:

I feel correlograms are now obsolescent. Their primary purpose was to allow workers to guess which models will fit the data best, but the speed of modern computers (at least in regression if not in time-series model-fitting) allows a worker to simply fit several models and see exactly how each one fits as measured by mean squared error. [The issue of capitalization on chance is not relevant to this choice, since the two methods are equally susceptible to this problem.]

This is not a good idea because the test of a model is supposed to be how well it can forecast, not how well it fits the existing data. In his three examples, he uses "adjusted root mean-squared error" as his criterion for the quality of the fit. Of course, over-fitting a model is going to make an in-sample estimate of error smaller, so his claim that his models are "better" because they have smaller RMSE is wrong.

In a nutshell, since he is using the wrong criterion for assessing how good a model is, he reaches the wrong conclusions about regression vs. ARIMA. I'd wager that, if he had tested the predictive ability of the models instead, ARIMA would have come out on top. Perhaps someone can try it if they have access to the books he mentions here.

[Supplemental: for more on the regression idea, you might want to check out older time series books which were written before ARIMA became the most popular. For example, Kendall, Time-Series, 1973, Chapter 11 has a whole chapter on this method and comparisons to ARIMA.]

  • $\begingroup$ The question is what are the (inherent) differences? $\endgroup$ – hbaghishani Jul 18 '13 at 11:25
  • $\begingroup$ As far as I can tell the author never described his home-brew method in a peer-reviewed publication and references to and from the statistical literature appear minimal and his main publications on methodological topics date back to the 70s. Strictly speaking, none of this “proves” anything but without enough time or expertise to evaluate the claims myself, I would be extremely reluctant to use any of it. $\endgroup$ – Gala Jul 18 '13 at 11:31
  • $\begingroup$ @hbaghishani the substantive difference is that autocorrelated data i.e within each series distorts the cross-correlative interpretation. Furthermore Gaussian Violations e.g. constant mean of the erros , constant variance over time , constant parameters over time need to be considered/rectified. $\endgroup$ – IrishStat Jul 18 '13 at 12:46
  • $\begingroup$ @flounderer People write textbooks to sell them and reap rewards. They sometimes include anachronistic methods that are being incorrectly taught because at some earlier date they were thought to be correct. To increase sales the publisher often demands ( from my personal experience ) outdated but incorrect methodology because these methods are in the syllabus. $\endgroup$ – IrishStat Jul 18 '13 at 14:07
  • $\begingroup$ @IrishStat modeling autocorrelated data could be done by dynamic regression models. Also, other models, like mixed models, could be used for such data. So, I don't think this feature is the substantive difference. $\endgroup$ – hbaghishani Jul 18 '13 at 14:43

Prof. E. Parzen, perhaps somewhat envious that he didn't propose the innovative methods of Box and Jenkins, suggested this approach of over-fitting and then stepping down. It fails for many reasons (many of which Flounderer has nicely summarized), including not identifying and remedying Pulses, Level Shifts, Seasonal Pulses and Local Time Trends. Additionally, changes in parameters over time or changes in error variance over time need to be considered.

I wrote a piece that you might be interested in. It is called "Regression vs Box-Jenkins" and is available from http://www.autobox.com/cms/index.php/afs-university/intro-to-forecasting/doc_download/24-regression-vs-box-jenkins

One comment about Darlington's procedure reflecting time, time*time, time*time*time*time as predictors. In the absence of Intervention Detection leading to the isolation of outlier effects, it is quite possible (and incorrect!) to conclude for higher powers of time. Beware non-statisticians performing statistical analysis as you would be wary of statisticians performing brain surgery. In fairness one might also add beware of non-time series statisticians/mathematicians trying to perform time series analysis with limited training in time series analysis.

Other posters (particularly whuber) on this list have repeatedly warned against using this "fitting approach", mostly in a univariate setting. This warning also applies to causal models.

Hope this helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.