This is a methodology question. I would like to make the data stationary but not transform it "too much" (information loss), before it is fit for statistical/ML purposes such as regression or PCA.

Typical procedure:

  1. We inspect the raw time series $x_t$. Assume we see trends or if the ADF test fails, so I seek to transform the data.

  2. We check that differencing $x_{t+1} - x_t$ results in stationary series. (Assume it does.)

  3. If possible, we seek alternatives to $x_{t+1} - x_t$: the transformation may excessively remove memory from the series. That may help stationarity but at the price of weaker correlations to other series. We therefore use fractional differencing, say with d=0.5, and check that the series is more stationary than the initial series.

Let's apply this to the following series:

Raw series

Conclude that it is stationary with $95\%$ confidence when using ADF implemented by adfuller() from Python's statsmodels package. The default parameter is $12(n/100)^{1/4}$ where $n$ is the series length. However, we see some (noisy) trend between 2011 and 2016. For the first half of the data the values are almost all negative. I conclude that the data doesn't give enough "labels" for model training.

I am therefore not sure whether the ADF test can be trusted. Would you feel comfortable using it for regression/PCA? Now compare with the fractional differencing with $d=0.5$:

Fractional difference

As expected this makes the series look more stationary. However, it also decreases the correlation to other similar dataset series transformed in the same way. Average correlation decreases as I increase $d$, as can be expected.

I therefore need to choose the smallest $d$ such that I feel comfortable that the transform is stationary. How do you know when to stop? I am reluctant to use the ADF pvalue as threshold to automatically set this considering the issue above.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.