0
$\begingroup$

I'm working with a data set and I am relatively un-versed in statistics / statistical terminology (my background is computer science) though I'm pretty techie and might know a bit more than some.

I'm working with a data set where I have a number of integer values (ranging from 1-100) and am counting the number of times each value occurs. These values tend to cluster over the total scale and have an average error of about .75 (rounding, basically) so they tend to fall into two consecutive (or sometimes three) buckets rather than a single one.

Here's an example of the data:

6 occurred 159 times in set - 33.9019 percent
5 occurred 133 times in set - 28.3582 percent
25 occurred 72 times in set - 15.3518 percent
24 occurred 69 times in set - 14.7122 percent
78 occurred 13 times in set - 2.77185 percent
76 occurred 13 times in set - 2.77185 percent
95 occurred 5 times in set - 1.0661 percent
97 occurred 4 times in set - 0.852879 percent

For example, in the set above both 5 / 6 are basically the same number (bucket) for my use and are very important due to the fact that they make up approx. 60% of the values. Similarly 24/25 are important with 30% overall. If there were a 23 or 26 it would also mix into that 30% bucket.

Values with less than 2-3% registration are basically noise so while the 76 / 78 are related (and probably useful) they're likely to be read as noise and thrown out which is ok but not ideal.

What I'm interested in is identifying the clusters over the overall range most efficiently and which ones contain the greatest number of samples.

Right now I'm using a brute force method of sorting by "value" and then adding buckets together which happen to be consecutive but I'm convinced there's probably some statistical method that will serve to "find and identify clusters"

Any leads / recommendations greatly appreciated!

$\endgroup$
0
$\begingroup$

Proof by example doesn't work, nor does statistical analysis method by example. You will have to come up with a precise formalization of what you need. On one-dimensional integer counts that will then likely yield an algorithm already.

I assume you want:

  • Buckets with less than min are not clustered.
  • Adjacent buckets with both at least min count go to the same bucket.

This leas to a trivial algorithm that just iterates over the array once, looking at bucket[i] and bucket[i-1].

$\endgroup$
  • $\begingroup$ Well, I've already done it all algorithmically... but it's very specific to my data set. It seemed to me that there must be a standardized statistical function/approach for identifying clustering in data? Seems like a signal-processing type problem? Again, I'm not a statistician so am trying to avoid recreating things which have already been done/established. $\endgroup$ – Jc Nolan May 22 '18 at 21:24
  • $\begingroup$ Finding adjacent non-zero integer values in an array is trivial. That is why there is no named algorithm for this. Clustering is multivariate, complex statistics. $\endgroup$ – Anony-Mousse May 23 '18 at 4:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.