I have a a data set containing of non-negative integer data for N subjects. In other words, each subject is represented by a vector of non negative integers (the vector length may vary from subject to subject, but is typically ~ 1000) and there are N such subjects. It is believed (hypothesized) that the distributions of the non negative values vary from subject to subject and the distributions are significantly different across an unknown number of groups (clusters) of subjects. My goal is to discover the hidden clusters and to learn the histogram bins that best distinguish the distributions across the clusters.

This seems to me to be a general problem that may have been well studied and solved in several fields. I'd appreciate if people could point me to relevant literature that shows how people have formulated and solved this problem. Thanks!

  • 2
    $\begingroup$ It isn't clear what role a "histogram bin" is intended to play in this analysis. Could you elaborate on that? $\endgroup$ – whuber Sep 23 '16 at 16:33
  • $\begingroup$ Also, what do mean by "optimal"? Given that there are no "optimal" clustering algorithms; they all require assumptions (e.g. Gaussian distribution, etc..) $\endgroup$ – Ray Sep 23 '16 at 18:15
  • $\begingroup$ @whuber Suppose I have two distributions A and B and bins b1, b2, b3 and b4 that I use for describing the empirical distributions of A and B. I am thinking of comparing b1(A) with b1(B), b2(A) with b2(B) etc. adjusted for multiple comparisons. By doing this I hope to learn whether A and B are different, and if so, what is the range of values where they differ most. $\endgroup$ – datameddler Sep 23 '16 at 19:06
  • $\begingroup$ @Ray Please see my response to whuber's question. By optimal, I mean the binning scheme that gives me the most power to detect the difference in two distributions. $\endgroup$ – datameddler Sep 23 '16 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.