Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 6 down vote favorite
2

I clustered my dataset of several thousand first-order Markov chains into about 10 clusters.

Is there some recommended way how I can evaluate these clusters and find out what the items in the clusters share and how they differ from other clusters? So I can make statement like "Processes in cluster A tend to stay in state Y once they get there, which is not true for processes in other clusters."

The transition matrices of those Markov chains are too large to just "look and see". They are relatively sparse, if that can help.

My idea was to take all the transition matrices in a cluster, sum them and plot it as intensity in a picture (in a scale from 0 to 255). Is there something more "professional" I should try out?

share|improve this question
Do you know that these processes are first-order Market chains (and, if so, how)? Assuming the answer to this is affirmative, then what additional a priori information do you know about the structure? I'm trying to identify why you are interested in clustering in the first place; I suspect knowing this will help our readers guide you more efficiently to a solution. – cardinal Sep 16 '12 at 21:57
The original data were clickstreams generated by users on my site. I created the markov processes so each process is to describe the clickstream of one user. I know there are books and papers that say that markov chains are inadequate for this, but my data do not include exact URL the user requested, just the "application" the URL belongs to. (My site is an Information system that is divided into 105 so called "applications" which are mostly self contained parts of the site, linked through a home page and side menu on every page) – user7610 Sep 17 '12 at 7:03
I am interested in clustering because I want to reveal groups of users that have similar patterns in using the site. I hypothesized that patterns that Markov chain captures is enough to distinguish such groups. I checked how the clusters I created correspond to roles users have on the site and it always looks the way that in a cluster there is a lot of users from one role and only a couple from other roles, so that looks promising. Hope that helps – user7610 Sep 17 '12 at 7:06

1 Answer

up vote 0 down vote

First, to get an idea, are your matrices of dimension 105 x 105, corresponding with the applications that you mention? When you say 'stay in state Y' does that mean stick around application Y?

Then, I would assume that outcomes such as "Processes in cluster A tend to stay in state Y once they get there, which is not true for processes in other clusters" are a bit too fine-grained with just 10 clusters. Have you tried a clustering of the application domain -- if I understand correctly you could cluster the 105 applications based on user behaviour. Next, have you looked at simple presence of users rather than transition, i.e. look at profiles of users across the 105 applications? It sounds as if you could use Pearson coefficient between user profiles; either on clusters of applications, or on the applications themselves. This could perhaps be extended towards transitions between applications, but currently I feel there is a huge mismatch between the number of clusters and the type of outcome you are interested in.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.