2
$\begingroup$

I am analyzing Twitter response data. I plotted a histogram shown below:

enter image description here

As you can see, most tweets get a response fairly quickly. I am confused whether this follows a power law distribution. I notice the J-shape of this distribution. It makes me think that this is indeed power law distributed.

However, power laws, like Zipfs law seem to "sort" the histogram in order to determine whether it follows a power law. Do we need to sort a histogram, before claiming it is a power law?

My reasoning that this follows a power law is that, it looks J-shaped and decreases rapidly. But I would like confirmation in my understanding.


I fit a one-parameter power law distribution and created a QQ plot below. The correlation coefficent is fairly low, so I'd presume that this isn't power law distributed then.

enter image description here

$\endgroup$
  • $\begingroup$ Could you please explain what you mean by "sort" a histogram? Note that this bar chart is not a histogram (because it uses a log scale for the counts). Note, too, that few statistical claims can be justified solely from a set of qualitative observations like "decreases rapidly:" you need to quantify the data behavior to draw clear and valid conclusions. $\endgroup$ – whuber Apr 4 '16 at 22:25
  • $\begingroup$ Thanks for the comment whuber. By sort, I mean: In Zipf's law: they sort the word counts by highest to lowest. Then it is claimed that this is power law distributed. My distribution follows a similar shape to a power law, but, I'm unsure of whether it is truly a Power Law (as I didn't sort the bars from highest to lowest). $\endgroup$ – looperr Apr 4 '16 at 22:28
  • $\begingroup$ Hey Guys: Do you have any ideas? This has been bothering me for sometime. Are both power law distributions? $\endgroup$ – looperr Apr 4 '16 at 23:09
  • 1
    $\begingroup$ @looperr: can you please quantify what the $x,y$ axis represent? What is "Twitter response data?" It looks like you're looking at response time to a post? The reason Zipf's law sorts words by frequency is that words inherently don't have a numeric order. Response time is already ordered so there's no real sense in reordering it by frequency. $\endgroup$ – Alex R. Apr 5 '16 at 0:01
  • 1
    $\begingroup$ Power laws are not that easily identified. See this paper by Shalizi, et al, which pretty much debunks the too frequent claims to power law distributions... santafe.edu/media/workingpapers/07-12-049.pdf In addition, there's this PDF of a presentation Shalizi gave which is more accessible ... stat.cmu.edu/~cshalizi/2010-10-18-Meetup.pdf $\endgroup$ – Mike Hunter Apr 5 '16 at 0:15
1
$\begingroup$

A power law is just a name for a particular form of distribution, specifically: $$ \begin{align} p(x) &\propto x^{-\alpha} \textrm{ } \textrm{ or}\\ p(x=X) &\propto x^{-\alpha} \end{align} $$ depending on whether the data are continuous or discrete. Despite a lot of hype, there's nothing particularly mystical or magical about them. Power law-distributed data has a long tail (highly right skewed) and falls along a (negatively-sloped) line on a log-log plot. Cosma Shalizi has a great blog post on determining whether your data is actually power law distributed. Importantly, he points out that many other distributions also share those properties (e.g., log-normal) and the usual trick of fitting a straight line to a log-log plot isn't particularly well-founded. In the post, he also links to some papers showing that many putative power law relationships aren't actually great fits to the data.

In brief, he recommends the following steps:

  1. Maximum likelihood estimation to find the scaling exponent ($\alpha)$
  2. The goodness of fit procedure described here to where the power law-y region begins
  3. A K-S test to check goodness of fit (using a bootstrap and not the regular tables because you're testing against an estimated fit).
  4. Vuong's test for model selection against other heavy-tailed distributions (e.g., log-normal)

For the specific case of Zipf's law, it shows that $\log(\textrm{rank}) \propto \log(\textrm{frequency})$ for many natural languages. If you want to compute this specifically, you will need to turn your data into ranks and frequencies, which seems like it will probably entail sorting it.

Most other power law relationships seem to describe a survival function (e.g., sales of books) and these also necessarily involve sorting. They don't, however, necessarily involve binning into a histogram: you can--and shoud--generate these directly from the data.

$\endgroup$
  • $\begingroup$ Hey Matt, I just updated the question with a QQ Plot for Power Laws. It seems to give me a low coefficent of determination! So - I suppose that this isn't a power law. Just to confirm, was using the QQ plot a good idea here? (i'm a novice stats guy) $\endgroup$ – looperr Apr 5 '16 at 0:54
  • $\begingroup$ That...looks like a pretty bad fit. You may want to work through the steps in that post anyway though, because many of the power-law things only "work" over a certain range. $\endgroup$ – Matt Krause Apr 5 '16 at 1:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.