Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have noticed that in some datasets that can be modeled using a power law distribution, there appears to be a kind of second tail trailing off at the lower end which appears as an almost horizontal line in log-log (for instance, see the figure below; source).
Is there any theory as to why this might happen?
To begin, let me say that the figure you cite is one that I generated back in 2005, so I can speak both to it specifically and to your more general question.
Let's start with that particular figure, which shows the complementary cdf (aka, the ccdf = 1-cdf =$\Pr(X> x)$) for the deaths or injuries or casualties (deaths+injuries) for terrorist events stored in the MIPT terrorism event global database. The Deaths data series will illustrate the point. Notice that the last value in that series is 2749. That event was 9/11. The next largest event in the database at the time I made this figure was about 400. Suppose there are $N$ events total in the deaths data set. The fraction of these events with severity $>400$ is $1/N$ and this is true for all $400<x<2749$ meaning that when we plot the value of the ccdf for all of those $x$ values, we get a horizontal line. Thus, the appearance of that line is just an artifact caused by there being no events in the data set with severity between 400 and 2749. (In subsequent versions of that figure, I only plotted the value of the ccdf at the $x$ values for $x$ actually in the data set, and so the horizontal line does not appear. For instance, see Figure 2 in this paper.)
More generally, this kind of behavior is not just common in heavy-tailed quantities (not just in power-law distributed ones) but actually expected. This is because the variance in these distributions is so large that any finite-sized draw from them will produce a maximum value that is well-separated from the body of the distribution. If we plot the value of the ccdf for all the values less than the maximum $x<\max_i \{x_i\}$, then we will get lots of these horizontal lines where there are no events with those sizes in the data set. And, these gaps only occur in the extreme upper tail (large values of $x$) because that is where the data is the sparsest. (In fact, for many of these distributions, including the power law, you can derive an expression for the expected value of the maximum in a draw of size $N$, as well as the value of the second maximum. So, if you wanted to, you could derive a mathematical expression for the size of these horizontal lines (if you knew the generating model).)
So, I would argue that there's nothing that interesting going on here, just standard stuff coming from visualizing high-variance distributions (on log-log axes).
