I'm working with data that is separated by year and classification and want to test if the data is correlated over each year. Essentially, I want to determine if the data is random each year or one year's data is correlated to the previous and possibly predictive of the future.

What statistical test is best for this?

For example: 1

Using this data, I would off-hand assume that there is correlation for users a & c. However, user b bounces between .4, .1 and .8 and thus would assume his yearly performances are fairly random.

Any and all help is appreciated! And let me know if any further clarification is helpful.


migrated from stackoverflow.com Jan 11 at 10:31

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ You should plot the time series data first then check the pattern present in the graph. Use ggAcf() function, it will show you whether data is correlated or not. $\endgroup$ – Zeeshan Jan 11 at 5:43
  • $\begingroup$ Plot an ACF plot for each user. $\endgroup$ – user2974951 Jan 11 at 10:34

What you are describing is known as serial correlation, where correlations exist between residuals at time t and t-1.

While plotting an ACF plot to determine the degree of autocorrelation across lags can help, you should formally test for this condition using the Durbin-Watson test, where:

H0: No serial correlation present

HA: Serial correlation present

The Durbin Watson statistic is calculated as follows:

durbin watson

Note that this is assuming your dataset is purely time series. If it is a panel dataset, i.e. both cross-sectional and time series, then you may need to conduct panel modelling instead, or at least test for serial correlation across the cross-sections separately.


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.