0
$\begingroup$

When dealing with a large volume of data and wanting to fit this data to five different bins (the number of bins is fixed), how might one find a good fit for the different bins relative to the data?

For instance, the data that I am currently using are classes with discrete frequencies. Like so:

class1: 10
class2: 382

In an example of the data I'm using there are 489 classes, an absolute sum of over 100,000 for the frequencies. A reasonable approach would be to delineate the bins for the histograms by the total frequencies/5 (which is 20,800 here). In an ideal world this might have 20 classes in the first bin, maybe sixty in the next, a hundred in the next and so on. However, because the data is exponential (a standard deviation of over 17,000), not even the first class would adequately fit into the bin (it has a frequency of 34,000).

What options do I have in order to create a decent fit for the different histogram bins?

Improve this question
$\endgroup$
2
  • $\begingroup$ You mention classes, as if your data were categorical, but you are asking about histograms, exponential data and standard deviation, all related to continuous data. Are your data categorical on numeric? $\endgroup$
    – Pere
    Dec 25, 2016 at 21:12
  • $\begingroup$ @Pere I asked this question before and used the word 'continuous' which worked out badly. I got a number of posters who went to some lengths to point out to me how integers are not actually continuous, and refused to address the actual question. To all intents and purposes though, you are correct. $\endgroup$
    – Stumbler
    Dec 26, 2016 at 15:09

1 Answer 1

Reset to default
1
$\begingroup$

If your data is made of 489 integer numbers, it's discrete - if we want to be strict - but it can be treated as continuous and therefore it's fine to draw a histogram.

A histogram with different interval sizes could be good to represent exponential data, although 5 bins for 489 data points seem too few to be very informative.

Anyway, if you want to select sizes for 5 bins, I'd suggest two ways:

  • Percentiles of your data (0-20%, 20%-40%, 40%-60%, 60%-80% and 80%-100%).
  • Percentiles of the exponential distribution if you know its rate (parameter) or you adjusted it from your data.

Anyway, if your data is actually exponential and sample size is large, both kinds of percentiles should be very similar.

If your question is about how to adjust an exponential distribution to your data, please clarify or ask again.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yes, both of those made sense. I was initially going to use the first way suggested. However the data doesn't really fit it. For instance the long tail means that about 40% of the individual classes/values share the lowest frequency, while in counterpoint only about 2-3% of the individual classes have the exceptionally high frequencies. I don't know much about rates of distribution - I'll research this more. The reason for 5 bins is to map ranges of frequencies to a fixed (low) number of variables for use in machine learning. $\endgroup$
    – Stumbler
    Dec 27, 2016 at 0:33
  • $\begingroup$ @Stumbler I don't know much about machine learning, but are you sure you can't use that data as numeric with your method? Some machine learning analysis methods like discriminant analysis work fine with numerical variables. If you shrink your numerical variable to just a 5 classes you are losing a lot of information. $\endgroup$
    – Pere
    Dec 27, 2016 at 0:56
  • $\begingroup$ Very true. Numerical data works well with some approaches. It doesn't happen to work well with Naive Bayes though - as the method would find it very difficult to find a correlation between any two numbers (1089 and 1090 would be considered as entirely different entities) $\endgroup$
    – Stumbler
    Dec 27, 2016 at 10:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.