I have noticed that in some datasets that can be modeled using a power law distribution, there appears to be a kind of second tail trailing off at the lower end which appears as an almost horizontal line in log-log (for instance, see the figure below; source).

enter image description here

Is there any theory as to why this might happen?

  • $\begingroup$ Is this not simply an artifact of discreteness? Are these counts or something similar? $\endgroup$ – Glen_b May 15 '14 at 19:34
  • $\begingroup$ I don't think that it has anything to do with quantization artifacts (though I don't know about the data set shown above, though 10e-4 is not even close to machine precision). It also happens when the actual counts are in the thousands, and the horizontal line becomes wiggly. $\endgroup$ – user32849 May 15 '14 at 20:31
  • 1
    $\begingroup$ Note that "deaths" and "injuries" are things that you count; why would this not be discrete? $\endgroup$ – Glen_b May 15 '14 at 22:41
  • $\begingroup$ I understand that. Again, this also happens when the number of counts is huge, so that the values along the horizontal line are NOT all the same. $\endgroup$ – user32849 May 15 '14 at 23:09
  • $\begingroup$ I didn't know what "wiggly" was intended to convey. I can't explain what I don't see. What's I see here are the blue values are multiples of $k=2\times 10^{-4}$, the red & black are similarly scaled integers (with different $k$). Show me what else you want explained and tell me what the data actually consist of (so if it's ratios, give numerators and denominators; ideally also plotted values) and I may be able to make some very good guesses. In the absence of either pictures or such explanations of what the data actually consist of, I have to wonder if you think we're telepathic or something. $\endgroup$ – Glen_b May 15 '14 at 23:13

To begin, let me say that the figure you cite is one that I generated back in 2005, so I can speak both to it specifically and to your more general question.

Let's start with that particular figure, which shows the complementary cdf (aka, the ccdf = 1-cdf =$\Pr(X> x)$) for the deaths or injuries or casualties (deaths+injuries) for terrorist events stored in the MIPT terrorism event global database. The Deaths data series will illustrate the point. Notice that the last value in that series is 2749. That event was 9/11. The next largest event in the database at the time I made this figure was about 400. Suppose there are $N$ events total in the deaths data set. The fraction of these events with severity $>400$ is $1/N$ and this is true for all $400<x<2749$ meaning that when we plot the value of the ccdf for all of those $x$ values, we get a horizontal line. Thus, the appearance of that line is just an artifact caused by there being no events in the data set with severity between 400 and 2749. (In subsequent versions of that figure, I only plotted the value of the ccdf at the $x$ values for $x$ actually in the data set, and so the horizontal line does not appear. For instance, see Figure 2 in this paper.)

More generally, this kind of behavior is not just common in heavy-tailed quantities (not just in power-law distributed ones) but actually expected. This is because the variance in these distributions is so large that any finite-sized draw from them will produce a maximum value that is well-separated from the body of the distribution. If we plot the value of the ccdf for all the values less than the maximum $x<\max_i \{x_i\}$, then we will get lots of these horizontal lines where there are no events with those sizes in the data set. And, these gaps only occur in the extreme upper tail (large values of $x$) because that is where the data is the sparsest. (In fact, for many of these distributions, including the power law, you can derive an expression for the expected value of the maximum in a draw of size $N$, as well as the value of the second maximum. So, if you wanted to, you could derive a mathematical expression for the size of these horizontal lines (if you knew the generating model).)

So, I would argue that there's nothing that interesting going on here, just standard stuff coming from visualizing high-variance distributions (on log-log axes).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.