How do I identify the "Long Tail" portion of my distribution?

Question

I have a number of series that would typically be described as normal skewed or Gamma distributed. For example, in a group of customers I may have calculated their spend over a fixed length of time. I then create a histogram of the distribution of spend and find an extremely long tail for the small group of high-spenders.

Since I want to identify these high spenders, my question is: are there methods to empirically inspect a distribution of values and approximate the point at which the distribution becomes "long-tailed" to create a cut in the data? I am not looking at inspecting a histogram to find the long tail but for a consistent method to systematically cut the data.

Answer 1

This question can be restated as: where do I cut a continuum to split it into two meaningfully different sets? And the answer is of course that there is no non-arbitrary way to do this. This is essentially the question of whether discretising continuous variables is advisable, about which we have many relevant threads:

Why is it Bad to Discretize a Continuous Variable?

When should we discretize/bin continuous independent variables/features and when should not?

What is the benefit of breaking up a continuous predictor variable?

Why should binning be avoided at all costs?

What is the justification for unsupervised discretization of continuous variables?

In short: it's almost never advisable to do this as part of an analysis.

But sometimes, a decision needs to be taken that requires some discretising. An example relevant to your situation would be: which customers should I spend money on by offering incentives (such as discounts) to? You could cut the distribution at some arbitrary threshold, as is done in the wikipedia article for 'long tail', and offer incentives just to people above this:

But this approach is weak without some justification. The good news is that for a decision like this, there's often more information you can and should take into account. The costs and benefits of the incentives are most important here, and should be used in any choice of threshold.