1
$\begingroup$

TL;DR: What is the impact of a linear trend on the correlation between time series that are (most likely) not spuriously correlated?


I'm currently trying to reconstruct/cross-validate an analysis delivered by one of my companies contractors.

The data is based on time series of sensor data (approx. 3.5m timestamps). Goal was to find the signals with the highest correlation with one specific signal.

Despite not being an expert in data science I was able to reproduce their data cleaning (drop columns with zero variance, interpolate linearly over smaller gaps, drop remaining columns containing NaN-values). But after that I'm not sure if I can confirm their findings.

Seemingly they did a simple pearson-correlation like

corr = df.corrwith(df['DesiredSignal'])

Yet looking at the data the signals seem definitely trended.

When I then apply a detrend-function like

from scipy import signal

df_d = signal.detrend(df[column])
df_n = pd.DataFrame(data=df_d)

and apply the corrwith-function to this new dataframe I get totally different results (e.g. a significant higher amount of highly negativ correlations).

My Question now is: Can I trust the findings of the contractor or are they rendered invaild by not considering the influence of trends on correlation or am I getting something completly wrong?

$\endgroup$
0
$\begingroup$

Correlation of non-stationary time series is a good place to start . Simple pearson-like cross-correlations can be easily computed BUT not easily interpreted when you have auto-correlated data.

Google has shut down enter image description here for a number of reasons ...including generating spurious false conclusions about cause and effect.

Consider Transfer Function model identification strategies when you have TIME-ORDERED DATA https://autobox.com/pdfs/regvsbox-old.pdf essentially filtering/adjusting for the autocorrelation with the series.

Also see Predict one variable based on another similar for an example of how two trending series were analyzed to extract the relationship between them. In this case the ordinary correlation coefficient would understate the relationship as a major intervention occurred due to an exogenous latent factor (level shift) .

$\endgroup$
  • $\begingroup$ Thank you very much for the detailed resources. They helped a lot and gave me new starting points for extended research $\endgroup$ – Viktor Katzy Sep 4 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.