I have a data set (24 x 1000) (hour x kwh) which contains 1000 time series of a buildings' power consumption, measured every hour. After applying k-means clustering using the dtw criterion I create 5 clusters as shown in the image below.


For a new day I am starting to collect values for the incoming time series. Hour 0 = 1.8 kwh, Hour 1 = 0.6 kwh, etc.

I want to create a model that will give me an indication from hour 0. That indication will show me how likely it is the incoming partial time series to belong in each cluster and it will change for every hour that I have a new incoming value. How would you approach this problem? If my descriptions is vague, please ask me anything. I am thinking a probabilistic solution...

Thank you.

  • $\begingroup$ I like eigen-calendars. $\endgroup$ – EngrStudent Apr 9 '18 at 18:06

One option:

For each cluster, estimate a distribution over time series, using the data assigned to that cluster as training points. This gives $P(X_t \mid C)$, where $X_t$ is a time series from $0$ to $t$, and $C$ is the cluster. You'll need a separate distribution for every value of $t$. You could use a simple multivariate Gaussian (probably with some constraints on the covariance matrix). Or, you could use some kind of temporal model.

Estimate the prior probability of each cluster as the fraction of points assigned to that cluster in your original data set. This gives $P(C)$.

Given a new time series $x_t$, the goal is to calculate the probability of membership in each cluster, $P(C \mid X=x_t)$. Use Baye's rule:

$$P(C \mid X=x_t) = \frac{P(X=x_t \mid C) p(C)}{P(X=X_t)}$$

You can just calculate the numerator for each class, then normalize so that the sum is 1.


Consider the cluster ids as class labels. Train your favorite probabilistic classifier on the original set of time series and corresponding cluster ids. You'll need to train a different classifier for each choice of $t$. Feed each new time series $x_t$ to the corresponding classifier, which will output the membership probabilities $P(C \mid X=x_t)$.


For different reasons (financial time series) I had the same problem and I solved the issue as the collegue above explained. The only weakness I see in this approach is that if you are studying a dynamic system also "the fraction of points assigned to each cluster in your original data set" changes over time, needing another level of dynamics on P(Cis) evolution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.