I have 15000 time series each of length 288 (corresponding to a sample taken every 5 minutes during a single day), and I want to group these time series by how similar they are based on shape and y-value across time. The example given below is an excellent match because, not only do the two time series have the same shape (go up and down in approximate sync), but also have very similar y-values across time.
The clustering method first generates cluster motifs - time series which are what a time series assigned to that cluster should ideally look like. The method then uses simple Euclidean distance to measure the similarity between the cluster motif and a time series and assigns a time series to a cluster if the distance falls below a specific threshold.
Changing parameters in the clustering method leads to different numbers of motifs and thus to clusterings with different numbers of clusters. As such, I have a clustering with 1759, 1090, 720, 483, 367, and 289 clusters respectively.
I want to have a way to evaluate the 6 clusterings, to say that one of the clusterings is "best". One way I have used is to calculate the average Residual Sum of Squares for each time series (compute the difference between the time series and the cluster motif of the cluster the time series is assigned to), and then take the average across all of the time series. As expected, as the number of clusters increases the average RSS goes down: intuitively this makes sense because each time series is more likely to be placed in a cluster where the match is better when there are more clusters to pick from.
I wanted to find another way of assessing the clustering where, the the number of clusters increases, the average value of this metric goes down. I have been reading about cross correlation and read through the answer here: How to use Pearson correlation correctly with time series
As far as I can tell the correct way to compute the cross correlation between a motif and a time series in the cluster is to difference the motif and the time series, and then compute the Pearson Product Moment Correlation on the resulting differenced time series.
If I compute the cross correlation for each one of the 15000 time series, and compute the average cross correlation for a given clustering, that will give me another way of assessing the 6 different clusterings.
Does this sound like the correct way to compute the cross correlation for the situation I have?
