# Measuring 'synchrony' with time series correlations

Question from a stats novice and StackExchange newb, an anthropologist with a computer science background. I'm looking for an appropriate statistical measure for 'synchrony' as in the following situation:

I have a panoramic video of several people sitting in chairs around the camera having a conversation for an hour. I split the video up by person and generated time series data for each individual, measuring how much they move, gesture, etc. over time (i.e. number of pixels different between successive video frames). So I have a graph of how still or animated a person is at each moment over an hour (using moving average to smooth things out).

I want to measure the 'synchrony' of the group at each moment, i.e. generate another time series that shows how 'synched up' their movements are at any given moment. What kind of statistical measure would be appropriate for this? Simpler/intuitive the better.

Some assumptions:

• If only one person is moving, synchrony should be near zero.
• If everyone is moving, synchrony should be near max.

Note that this will be used primarily for visual inspection and analysis, to highlight those parts of the video where people are synchronous.

-Daniel

• ps: I'll give credit for the best answers in my ph.d. dissertation. Dec 7, 2011 at 0:52
• Why not just average the individual time series? Dec 7, 2011 at 21:27
• If nobody moves very much, do you want to give that as a high a value as if everybody is moving a lot? Or do you want to weight large movement synchrony higher than small movement synchrony? The first could be done with a rolling correlation, the second be done by maybe multiplying the two time signals together and then weighting that by the correlation. I'd check some literature first to make sure that these ideas are valid. Dec 8, 2011 at 15:14
• @jbowman Averaging is a decent first pass. But it gives too large a value when only one person is moving, especially when the group is small. I want to discount occasions when only one person is moving but highlight occasions when at least two people are moving. Dec 8, 2011 at 20:09
• @MattAlbrecht Great question. The generated time series should give low values when nobody is moving much. (But you make me realize that the rolling correlation would also be useful!) For your multiplication suggestion, do I normalize the data to [0.0, 1.0] or stay in the range of positive integers (number of pixels)? Dec 8, 2011 at 20:17

Ok interesting question. Think I know a proper answer use Ramseyer and Tsachers model/method (Nonverbal Synchrony or Random Coincidence? How to Tell the Difference).

Your data seems excellent for it! Below a short description by head, might have some mistakes here so please read the referred papers as well.

They use Motion Energy Analyses (frame to frame pixel difference after passing some filters to exclude high-frequency lighting influences e.g. Bandworth filter as done by Paxton and Dale (to be published))

This is followed by Pearson's cross-correlation using different time-lags (gives you info on leading following behvaiour as well) (and followed by a peak finding algorithm). Then you can use a statistical analyses by testing it compared to 99 fake (time-shifted dyads). This will give you enough information for creating a synchrony-rating. Please send me a link if you publish it, wonder what you will find always good to have examples of performed studies.

Problem will be on the multi-person bit. My PhD focuses on this and a real-time measure to be used in interactive entertainment systems (instead of using the 99time shifted ones), haven't found solutions for that, could use multiple comparisons for the time being. If you want more info read into Emilie Delaherche's work as well, she gives a nice overview. Boker, Grammer Ramseyer Tsacher, they all provide more info on the peak picking, cross correlation etc.

• Great article. Thank you. Like the authors, I used Motion Energy Analysis (frame differencing). The multi-person cross-correlation was also a problem point. For the dissertation, I ended up just visualizing the number of people moving Aug 13, 2012 at 23:48
• ...For the dissertation, I generated a time series of the number of people moving at any given moment (above a minimum threshold). I then made simple probability calculations: e.g. if no one is currently moving, what is the likelihood that if someone starts moving that others will move also? In general I found that about 90% of the time, others will move too. For publication, I'd like to do more sophisticated stats and these references are really helpful. Aug 13, 2012 at 23:54

Well, there are established measures for synchronization. There even is synchronization based clustering. Why don't you just use these measures?

Read up on ''Kuramoto model'': http://en.wikipedia.org/wiki/Kuramoto_model

• I think I am using the term 'synchrony' in a different sense than the Kuramoto model assumes. I don't think human bodies aren't adequately modeled as periodic oscillators. A closer model might be Per Bak's sandpile or other dynamic cascade. But in any case, I'm not trying to create a model that corresponds to my data. I'm adding an analytic layer to reveal patterns in it. Dec 8, 2011 at 20:26

Here's what I was suggesting in R code. I don't know what software you're working with, but at the very least you can download R for free and run the script pretty easily just to see what I was talking about and then create your own version. If you were in R, a lot of the loops could be replaced with the "rollapply" function in the "zoo" package. But this way the code is self contained.

What I've done is created three time series: 1. Simple signal, where the next value at point (i+1) correlates with the preceding value (i). 2. Signal based on the first so they should correlate highly but have different amplitudes 3. A random signal made with Gaussian noise

What this does leave out is phase shifts, that is, if a person a starts moving and then person b starts moving after but in time with, this method will underestimate the correlation. This can be rectified by including a number of time shifts. Or, possibly by increasing the length of time for the rolling average (acts like a low pass filter).

Of course there are other methods which might be more suitable, but those are based on oscillating signals, e.g., you could use wavelets for a time-frequency decomposition and then calculate a between person phase-locking across a number of frequencies. Then create phase and coherence maps. If you think this might be more what you're after, I have scripts for that too, but you may want to look into dedicated packages in matlab or R.

Before applying, you'd probably need to take a couple of random samples from your videos, or perhaps even a "training" video and see what parameters give you the information that you're looking for. Then apply this to your actual samples. E.g., changing the length of the rolling average, playing with phase shifts, adjusting the weighting parameter. You could even get boostrapped CIs if you wanted.

Here's the R script:

#highly correlated series
rl<-20                  #rolling average length
x<-5                    #Just a starting value
xvec<-3000
#1st time series, made so that the next value correlates with preceding value
for(i in 1:(xvec-1)) { x[i+1] <- x[i] +rnorm(1, 0, 0.2) }
y<-x+rnorm(xvec, 0, 0.3) #Second series based on 1st series for high correlation
xy<-(x*y)/max(x*y)      #For weighting

#Calculating rolling correlation with 20 values either side
cxy<-sapply((rl+1):(xvec-rl+1), function(i) cor(x[(i-rl):(i+rl)], y[(i-rl):(i+rl)]))
#Smoothed rolling correlation by rolling average
cxym<-sapply((rl+1):(xvec-3*rl+1), function(i) mean(cxy[(i-rl):(i+rl)]))
#Smoothed weighting
xym<-sapply((2*rl+2):(xvec-2*rl+2), function(i) mean(xy[(i-rl):(i+rl)]))

par(mfcol = c(2,2))     #Create plot so that there are 4 figures per plot space
plot(1:xvec, x, type="l"); lines(1:xvec, y, col=2)  #plot 1st and 2nd time series

#Plot correlations
plot((rl+1):(xvec-rl+1), cxy, type="l", xlim=c(0, xvec), ylim=c(-1,1))
lines((2*rl+2):(xvec-2*rl+2), cxym, col=2)      #Smoothed rolling correlation
lines((2*rl+2):(xvec-2*rl+2), cxym*xym, col=3)  #Smoothed weighted correlation

#No correlation between series and plot
y<-rnorm(xvec, 5, 1)
xy<-(x*y)/max(x*y)
cxy<-sapply((rl+1):(xvec-rl+1), function(i) cor(x[(i-rl):(i+rl)], y[(i-rl):(i+rl)]))
cxym<-sapply((rl+1):(xvec-3*rl+1), function(i) mean(cxy[(i-rl):(i+rl)]))
xym<-sapply((2*rl+2):(xvec-2*rl+2), function(i) mean(xy[(i-rl):(i+rl)]))
plot(1:xvec, x, type="l"); lines(1:xvec, y, col=2)
plot((rl+1):(xvec-rl+1), cxy, type="l", xlim=c(0, xvec), ylim=c(-1,1))
lines((2*rl+2):(xvec-2*rl+2), cxym, col=2)
lines((2*rl+2):(xvec-2*rl+2), cxym*xym, col=3)


A dynamic spontaneous synchronization type of visual would be useful here. Please see the example of firefly flashing simulation using star logo.

http://skyeome.net/wordpress/?p=56

http://education.mit.edu/starlogo/

You could use some measure of a member's pixel difference between frames (mean of difference in intensities?) as the sequential step measurement of an individual's movement; then taking a sequence of training samples on the first few minutes of footage, find a reasonable estimate of the absolute range of min and max movements among all individuals.

From there, you could quantize the range into some set of levels with a visual intensity dot corresponding to an individual's movements over time. All individual member dots would be plotted in a cluster as in the attached image. From there you could generate the rolling synchronisation plot as in Figure 2. Possibly using some type of kernel density related to the frequency of each of the quantized levels per each time slice on the rolling plot. If the level of all members were all aligned then the bin width would be minimized and intensity maximized; any lower correlation would result in larger dispersion and lower max intensity of the density snapshots.

I think it could be as simple as plotting the median activity level for the three participants. It wouldn't go up much just because one participant became active, but would go up much more if two or three participants were active.

You should use running correlations for pairs of individuals. Here's an example:

Corbetta, D., & Thelen, E. (1996). The developmental origins of bimanual coordination: a dynamic perspective. Journal of Experimental Psychology: Human Perception and Performance, 22(2), 502-522.

You can do it easily on excel. Mail-me if you have difficulties doing this.

For multiple oscillators (i.e., persons) the Kuramoto Model, using cluster phase. It is a lot more complicated. Probably you will need a Matlab routine for this.