How to divide ordinal set into bins?

Question

What is the best way to divide Cross Validated/Stack Overflow/Server fault reputations into separate bins, considering reputation systems tend to be highly skewed? For example: Suppose I want a set of 64 bins - 0-250,250-500,500-100...10,000-11,000. Is there a method that takes into account the number of users that have a specific reputation wen specifying the bin limits or can I just assign arbitrary bins?

slotishtype

Answer 1

One approach would be to use v-optimal histograms, which are histograms that select bin sizes in order to minimize the sum of squared errors of the representation. Another alternative is equi-depth histograms which may be useful if you'd like each bin to contain (approximately) the same number of elements.

As Whuber mentioned, it would be helpful to first identify what it is that qualifies as a "good" binning.

From the revised question, it sounds like slotishtype is looking for an equi-depth histogram. Here is some example Matlab code for computing the bins, hope this helps.

% INPUT: 
% v     input vector of values (sorted)
% n     number of bins youd like to split the data into
%
% OUTPUT: 
% domain    provides the representative value for each bin
% rep       provides the range of each bin
%

% sample input from a skewed distribution
v = sort(exprnd(1, 1000, 1));
% number of bins
n = 5;

N = length(v);
d = zeros(N, 1);

[sv ix] = sort(v);
% figure out how many elements we'll put into each bin
bin_size = round(N / n);
rep = zeros(n, 3);

s = 1;
for i = 1:n
    e = min(N, s + bin_size - 1);
    % compute the representative of this bin 
    % (I use the mean, but you could change this depending on your problem)
    rep(i, 1) = mean(sv(s:e));

    % this is just to show the range of values each bin takes on
    rep(i, 2) = min(sv(s:e));
    rep(i, 3) = max(sv(s:e));

    % d builds up a representation of the original data using the bin
    % representatives for each value (vector quantization)
    d(ix(s:e)) = rep(i, 1);

    s = e + 1;
end
domain = unique(rep(:, 1));

fprintf('Bin representative vlaues:\n');
rep

fprintf('Range of each bin:\n');
rep(:, 3) - rep(:, 2)