2
$\begingroup$

What is the best way to divide Cross Validated/Stack Overflow/Server fault reputations into separate bins, considering reputation systems tend to be highly skewed? For example: Suppose I want a set of 64 bins - 0-250,250-500,500-100...10,000-11,000. Is there a method that takes into account the number of users that have a specific reputation wen specifying the bin limits or can I just assign arbitrary bins?

slotishtype

$\endgroup$
3
  • 3
    $\begingroup$ Could you provide additional context about this? E.g., do you work with ordinal data such as Likert-type item or discrete scale scores that are are highly skewed, where you want to find some kind of meaningful quantiles on the underlying scale? What's the purpose of this discretization? $\endgroup$
    – chl
    Dec 16, 2010 at 13:25
  • $\begingroup$ Thanks for the reply, Let me re-work the question...I am not sure of the correct way to communicate the question and the correct vocab to use... $\endgroup$ Dec 16, 2010 at 14:29
  • 1
    $\begingroup$ The question as it stands is clear enough (although I am wondering how you can characterize merely "ordinal" data as "skewed"). What we need is a statement of your objectives and how to determine how well any particular binning will meet them. That is, why are you binning? How can that influence the decision or action you ultimately will take based on the data? It can also help us to understand something about the data generation process: are these data a random sample of something? A convenience sample? An entire population? $\endgroup$
    – whuber
    Dec 16, 2010 at 16:48

1 Answer 1

4
$\begingroup$

One approach would be to use v-optimal histograms, which are histograms that select bin sizes in order to minimize the sum of squared errors of the representation. Another alternative is equi-depth histograms which may be useful if you'd like each bin to contain (approximately) the same number of elements.

As Whuber mentioned, it would be helpful to first identify what it is that qualifies as a "good" binning.

From the revised question, it sounds like slotishtype is looking for an equi-depth histogram. Here is some example Matlab code for computing the bins, hope this helps.

% INPUT: 
% v     input vector of values (sorted)
% n     number of bins youd like to split the data into
%
% OUTPUT: 
% domain    provides the representative value for each bin
% rep       provides the range of each bin
%

% sample input from a skewed distribution
v = sort(exprnd(1, 1000, 1));
% number of bins
n = 5;

N = length(v);
d = zeros(N, 1);

[sv ix] = sort(v);
% figure out how many elements we'll put into each bin
bin_size = round(N / n);
rep = zeros(n, 3);

s = 1;
for i = 1:n
    e = min(N, s + bin_size - 1);
    % compute the representative of this bin 
    % (I use the mean, but you could change this depending on your problem)
    rep(i, 1) = mean(sv(s:e));

    % this is just to show the range of values each bin takes on
    rep(i, 2) = min(sv(s:e));
    rep(i, 3) = max(sv(s:e));

    % d builds up a representation of the original data using the bin
    % representatives for each value (vector quantization)
    d(ix(s:e)) = rep(i, 1);

    s = e + 1;
end
domain = unique(rep(:, 1));

fprintf('Bin representative vlaues:\n');
rep

fprintf('Range of each bin:\n');
rep(:, 3) - rep(:, 2)
$\endgroup$
5
  • $\begingroup$ Thanks guys, sorry for not explaining myself more clearly. I amended the question above...see above $\endgroup$ Jan 20, 2011 at 10:10
  • $\begingroup$ By the way, I would up-vote the comments but my rep is too low. $\endgroup$ Jan 20, 2011 at 10:23
  • $\begingroup$ some R example code for your solution would echo far. :) $\endgroup$ Jan 20, 2011 at 10:46
  • $\begingroup$ @Roman Luštrik, I added some sample code in Matlab for equi-depth histograms (I'm not a regular user of R), hope this will suffice! $\endgroup$
    – Nick
    Jan 20, 2011 at 18:07
  • $\begingroup$ Effort is always a plus in my book. $\endgroup$ Jan 21, 2011 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.