I am working on clustering of data streams. For my purpose Sequential Leader Clustering (SLC) have given fair result, as i need not give number of clusters (like k in k means), which require some prior knowledge about data, and this is not affordable in Real time data streams ( The algorithm need to be strictly one pass over data).

Now, I'm looking forward to extend this method for distributed systems (such as map-reduce framework). The data streams is being collected by multiple nodes.

Can i just run SLC in individual nodes, and combine the result without much more difficulties. Or there is some other trick that i can apply for this.

Or is there any other algorithms, that i need to look for?

  • $\begingroup$ What kind of data are you working with? You might look into min-hash and other kinds of locality sensitive hashing. $\endgroup$ – Dave31415 Mar 2 '14 at 21:49
  • $\begingroup$ Thanks for the answer... I am working with the streams of bitstrings (256 bits). The similarity measure for this is similar to jaccard similarity, i.e. # of 1's in (ANDed of 2 bitstrings)/ # of 1's in (ORed of 2 bitstrings). As far as i have understood, i cannot permute to the different combinations of min-wise hashing (because the elements in the strings are just 1 or 0). So , is there is any hashing for this similarity and data type??? $\endgroup$ – bistaumanga Mar 20 '14 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.