I have the following datasets of several variables:

  • hourly data of the year 1990
  • hourly data of the year 2000
  • hourly data of the year 2010

Now, I was planning to select and estimate a linear regression model based on the dataset of the year 1990 and then validate/check its accuracy by applying the model to the other two datasets and compare the dependent variable based on the regression model with the actual dependent variable of the dataset.

Is this approach correct? If so - am I supposed to completely ignore the two validation datasets? By that I mean not even looking at it and doing descriptive analysis?

Note: All the datasets contain the same variables.


Is this approach correct?

It depends on the data, but most likely the answer is "no this is not correct". There are very likely trend effects which would 2010 and 2000 different from 1990.

You might want to identify a trend between the 3 years, and then use that to de-trend the data, but that would only work if the trend is linear and even then you are making some strong assumptions, because of the large gap between the 3 samples.

Since you are dealing with a year's worth of hourly data, there are likely multiple seasonalities within your data (daily patterns, weekly patterns and monthly patterns). It depends on your data, but it might possible to assume that these patterns haven't change over the years. You might be able to normalize your data relative to yearly averages. Then you would be able to use 2000 and 2010 for validation.

Consider the following time series of international airlines passengers shown below:

The seasonality for 1954 and 1959 are very similar, but the level is much higher for 1959 because of the visible trend in the data. So if you trained a model on 1954 data and then tried to validate it using 1959 data, it would be completely off.

But if you were to de-trend the data or normalize it to the yearly average, then a model trained on 1954 data might do a better job at forecasting 1959 data.

enter image description here

  • $\begingroup$ Thank you for your answer. Yes there are multiple seasonalities within the data. Yet, I do not understand why I should identify a trend between the years? I was of the opinion that the "buffer" between the years is long enough to be able to treat them as seperate samples of the same data. $\endgroup$ – shenflow Dec 12 '17 at 18:28
  • $\begingroup$ Suppose the linear regression model includes seasonal dummies as well as a linear trend. How is a trend between the datasets prohibiting me from using one for construction and estimation of the model and the other two for comparison/validation of the models accuracy for a specific relationship between given variables? $\endgroup$ – shenflow Dec 12 '17 at 18:30
  • $\begingroup$ I understand what you mean - the data might be non stationary and thus, display different behaviour and different points of time. However, I include seasonal as well as trend dummies in my regression model. To my understanding this implies, that, as long as the seasonality/trend structure remains the same, the model should be applicable. (I thought that including seasonal dummies/trend is essentially the same as detrending/deseasonalizing data.) $\endgroup$ – shenflow Dec 12 '17 at 19:04
  • $\begingroup$ Regardless of that - if the trend/seasonality would change and make the model inappropriate over the years, why should I not be able to use the other two datasets as test sets? The result might be, that there are significant differences between model/actual values for the test sets. This would indicate an inaccurate model for the data. This has nothing to do with my initial problem, which was: Can I use one of those datasets for training and the other two for testing? $\endgroup$ – shenflow Dec 12 '17 at 19:07
  • $\begingroup$ I am not implying that I have a correct model. I just want to know if my "validation" process is correct in a statistical way, expressed differently: If I can seperate the given datasets like I explained. $\endgroup$ – shenflow Dec 12 '17 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.