10
$\begingroup$

Imagine the following problem. I have weekly snapshots of price data of K items, as well as of various features/predictors. I want to predict how much the price will change 2 years from now.

I assemble my dataset as follows: each row consists of features for each item for each week, and the output variable is forward 2 year price returns. The date of the observation is not in my dataset - I only use it to separate the dataset into a training and validation set, i.e. in cross-validation (where I discard 2 years of data before and after the validation time-period (which is 1 year) to prevent data snooping).

Clearly, the samples from two consecutive weeks for the same item (and even for different items) will be highly correlated, both in terms of the features and in terms of the response variable (as the forward 2 years will largely overlap, meaning the returns will be very similar). What potential problems can it cause for supervised learning approaches, e.g. random forests or gradient boosted trees?

My thoughts are:

  1. The effective size of the dataset will be smaller than expected. I.e. my dataset of, say, 100'000 observations will behave more like a dataset of 100'000 / (52*2) ~= 1000 observations, because that is the number of samples for which response will not have autocorrelation. That will significantly affect the complexity of models that I can fit to the data, i.e. I will have significant overfitting problems and have much poorer results than expected.
  2. Because of features being very near each other on consecutive weeks for each item in the feature space, my dataset will cover the feature space a lot worse than expected, again reducing the "effective" size of the dataset.
  3. Using only 1 year of data for validation in cross-validation will result in high variance of cross-validation results, because once again, the effective number of samples in the validation set will be ~K rather than 52*K.

Are these valid concerns? If yes, does it mean that with K~=100, I will need hundreds, if not thousands years of data to be able to train a reasonably complex non-linear model from hundreds of features, e.g. using random forests or gradient boosted trees? Or am I being over-pessimistic and my argument about "effective dataset size" above is nonsensical?

$\endgroup$
6
+50
$\begingroup$

You touch on an issue that has a parallel in the econometric literature. It's called the long-horizon predictability problem. While it's difficult to predict the stock markets and currencies in the short term, some econometric studies have shown that long term returns are "much more predictable" using covariates like dividend yields.

Well, it turns out there is a subtle flaw in these models. Since both the response and the predictors cover an overlapping period, they're highly autocorrelated across horizons, and the data points are not independent.

Here is a couple of papers I could find in my library. The Berkowitz paper is probably the most devastating on the subject.

A study that shows long horizon predictability :

Mark, N. C., & Choi, D. Y. (1997). Real exchange-rate prediction over long horizons. Journal of International Economics, 43(1), 29-60.

Criticism and statistic tests :

Berkowitz, J., & Giorgianni, L. (2001). Long-horizon exchange rate predictability?. The Review of Economics and Statistics, 83(1), 81-91.

Boudoukh, J., Richardson, M., & Whitelaw, R. F. (2006). The myth of long-horizon predictability. The Review of Financial Studies, 21(4), 1577-1605.

Richardson, M., & Smith, T. (1991). Tests of financial models in the presence of overlapping observations. The Review of Financial Studies, 4(2), 227-254.

$\endgroup$
3
  • 2
    $\begingroup$ Thank you. A quick look through the literature (including Mark's response to Berkowitz's criticism) suggests that the question of whether long-term returns are "more predictable" is not quite settled. It looks like I should not be too worried about spurious correlations if I use cross-validation correctly, as long as I do not unconsciously introduce feature selection from the whole dataset into the model. But it is not much clearer to me yet whether sample size will be a problem or not. $\endgroup$ – rinspy Jul 31 '17 at 9:19
  • $\begingroup$ @rinspy Glad you read up on topic. It's been a while but my impression is Berkowitz et al has a much stronger case and Mark's defense not convincing, thus you haven't seen more papers on "long horizon predictability" of this and that. Now back to your question. I'm not sure what you mean by using CV "correctly", please elaborate. $\endgroup$ – horaceT Jul 31 '17 at 17:25
  • $\begingroup$ @rinspy Root of the problem is data independence. Almost all the machine learning algorithms rely on this fundamental assumption. If you take two data points that measured overlapping stuff, the two data points are not independent. The only algorithm that I'm aware of that does not require this assumption is reinforcement learning. $\endgroup$ – horaceT Jul 31 '17 at 17:30
3
$\begingroup$

Let's sketch your problem as:

$$ f(\{X_t: t \leq T \}) = X_{T+1} \tag{1}$$

that is, you are trying to machine learn a function $f(x)$. Your feature set is all the data available until $T$. In a somehow overloaded notation I wanted to highlight the fact that if we look at $X$ as a stochastic process, it'd be convenient to impose that $X$ is adapted to a filtration (an increasing stream of information) - I'm mentioning filtrations here for completeness sake.

We can also look at equation $1$ as trying to estimate (here):

$$ E[X_{T+1} | X_T, X_{T-1}, ..] = f(\{X_t: t \leq T \}) $$

In the simplest case that pops in my head - the OLS linear regression - we have:

$$ E[X_{T+1} | X_T, X_{T-1}, ..] = Xb + e $$

I am suggesting this line of thought to bridge statistical learning and classic econometrics.

I am doing so because, no matter how you estimate (linear regression, random forest, GBMs, ..) $ E[X_{T+1} | X_T, X_{T-1}, ..]$, you will have to deal with the stationarity of your process X, that is: how $ E[X_{T+1} | X_T, X_{T-1}, ..]$ behaves in time. There are multiple definitions of stationarity that try to give us a flavor of time-homegeneity of your stochastic process, i.e how the mean and variance of the estimator of your expected value behave as you increase the forecasting horizon.

  • In the worst case scenario, where there is no sort of homogeneity, every {X_t} is drawn from a different random variable.
  • Best case scenario, iid.

We are in-between the worst and best case scenario: autocorrelation impacts the type of stationarity a stochastic process displays: the autocovariance function $\gamma(h)$, where $h$ is the time gap between two measurements, characterizes weakly stationary processes. The autocorrelation function is the scale-independent version of the autocovariance function (source, source)

If the mean function m(t) is constant and the covariance function r(s,t) is everywhere finite, and depends only on the time difference τ = t − s, the process {X(t), t ∈ T} is called weakly stationary, or covariance stationary (source)

The weakly stationary framework should guide you on how to treat your data. The key takeaway is that you can not put auto-correlation under the rug - you have to deal with it:

  • You increase the granularity of the time mesh: You throw away data-points (less granularity and a lot less data to train your model) but auto-correlation is still biting you on the dispersion of $E[X_{T+1} | X_T, X_{T-1}, ..]$ and you'll see lot of variance in your cross-validation

  • You increase the granularity of the time mesh: sampling, chunking and cross-validation are all much more complex. From a model point of view, you'll have to deal with auto-correlation explicitly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.