Correlation between two binary time series

Question

I have two binary time series, say df1 and df2. For each 'day' there could be 1 if the user had an activity that day, 0 otherwise. I want to compare the two time series (for two different users) to understand if there is a correlation between the two.

First question: does it make sense to use pearson correlation for binary series? if not, what's best?

Second question: my activity time series could have some lag, meaning that user 1 can have an activity on a day, but user 2 may respond the next day. How can I best catch this behaviour? If I use a lag of +1 day, I may not catch the 'same day activity' correlation.

Answer 1

First question: does it make sense to use pearson correlation for binary series? if not, what's best?

No. The Pearson correlation coefficient normally applies to continuous variable.

I suggest that you use chi squared test for independence, which operates on a contingency table $2\times2$ where you just accumulate frequencies of a combination of outcomes for user1 and user2.

One important step is making sure that outcomes in each of your datasets are accumulated independently, i.e., IID assumption holds.

How can I best catch this behaviour?

What about trying several lags for both users and getting the chi squared statistics related to them? The highets statistics would tell you which "best" lagged model of dependence you have found.

Answer 2

Complementing the previous answer. Using ADWIN2, an adaptive windowing algorithm, it is I think possible to adaptively select windows with a constant value for a metric.

Meaning that this algorithm is able to select windows over which the chi square is constant, and starting a new window if the chi square has changed; in this case you can monitor the correlation using a streaming algorithm.

Answer 3

Since you work with time series, you could use the cross-correlation function between the two series. In this accouts for shifted sequences, i.e. a time lag as well. You could normalize them with the product of standard deviations of each signal and arrive at a lag-dependent pearson coefficient. The approach is wide-spread in digital signal processing and is also used on binary sequences.

Edit to clarify:
If both sequences have the same length N, you can represent them as vectors with the same dimensionality N. then you can calculate the scalar product between them, normalize them with their euclidic length and have a pearson correlation coefficient. Since they are binary, the length is just the square root of the number of ones in each vector.

You can now introduce a time lag between the two sequences, when zero padding to both vectors is allowed.

The result is the cross correlation function (numpy example).

The zero padding has an effect to the similarity measure, but you should still be able to see a possible similarity spike if the lag variable is small enough.

If you assume some periodicity, it might even make sense to use a circular correlation which can be implemented with a cyclic convolution scipy.ndimage.filters.convolve1d (See https://stackoverflow.com/questions/37150672/circular-convolution-in-python for source) by flipping one of the vectors first. The result does not have the problem from zero padding, but is not justified very well if there is no periodicity.

By the way, what result do you expect when only one of both users is active? My naive idea of directly using bit vectors for the correlation would lead to a 'similarity' of 0 here. Therefore it might be wise to apply a mapping of 0 -> -1 first to get a negative result in this case. Furthermore, the normalization factor becomes just 1/N.

Answer 4

you can use decay, if today happens , value to be 1, tommorrow, if not happen , can be 0.9, day after tommorrow, if still not happen , 0.81, use exponential decay, then use pearson coefficient, it may capture some dependence

Answer 5

The Jaccard similarity index of the two sets might be useful here, as a way of comparing the degree to which two individuals are active on the same days as each other. It wouldn't account for lag though, and you would need to use one of the other measures suggested.