What is the best way to find change points in empirical power law distributions?

Question

I'm looking at some data that conform reasonably well to a continuous power law distribution, according to a Kolmogorov-Smirnov test that compares the estimated power-law fit to the data (per Clauset, Shalizi, and Newman (2009), "Power-law distributions in empirical data," SIAM Review). The problem I'm facing is that I'm not exactly sure what current best practice is for looking for change points for the slope coefficient of that distribution.

Most change-point algorithms, as I understand it, aren't designed to work with long-tailed distributions, in which sample means converge only very slowly to population means. I can't estimate slope coefficients via OLS and then compare the coefficient on one side of the proposed change point vs. the coefficient on the other because the ranking procedure produces serial correlation among residuals, which means that the standard errors can be grossly understated (Gabaix and Ioannides (2004), "The Evolution of City Size Distributions," in Henderson and Thisse, Handbook of Regional and Urban Economics, volume 4). And the Hill MLE estimator is troublesome in finite samples (in the smallest sample, I'm looking at an n of under 100).

I could perhaps (?) bootstrap the coefficients to get the standard errors right or put together a likelihood ratio test, but I'm starting to wonder whether I'm making the problem bigger than it is. Couldn't I just use a two-sample Kolmogorov-Smirnov test to calculate the probability that the distributions on either side of the proposed change point differ? That strikes me as fairly intuitive and reasonably safe, but I'm coming to the realization that long-tailed distributions are the graveyard of statistical procedures that seem intuitive and safe. Since the empirics of power-law distributions seems to be an area of active development, I thought it would be best to consult the hive mind. Thanks in advance.

Answer 1

Couldn't I just use a two-sample Kolmogorov-Smirnov test to calculate the probability that the distributions on either side of the proposed change point differ?

No, it doesn't tell you this probability*. You appear to be interpreting a p-value as the probability the alternative is true. It's not.

*(it's also not clear to me how your proposal would work even if it did)

---

Unless you have very solid theoretical reasons for searching for a piecewise power law, I'd suggest you seriously consider abandoning the idea. If you persist in it, I'd advise you follow the lead of your reference in looking at Vuong's test for alternatives.

One of the authors you refer to in particular disparages the idea multiple times.

For example, here's Shalizi in a post titled "So You Think You Have a Power Law — Well Isn't That Special?":

Lots of distributions give you straight-ish lines on a log-log plot. [...] Don't even begin to talk to me about log-log plots which you claim are "piecewise linear".

(emphasis mine)

Also see this talk, which points out that the reference you mention:

[10] looked at 24 claimed power laws [...] Of these, the only clear power law is word frequency

The immediately following slides (titled "What's Bad About Hallucinating Power Laws?" and "Does It Really Matter Whether It's a Power Law?") go on to explain why referring to something that's vaguely consistent with a power law as a power law is a problem and why you probably don't need to do it.

Indeed, to my eyes these concepts are at least suggested in the paper you mention (though more subtly expressed).

---

To return to the matter at hand:

• You can posit piecewise power-laws (consisting of several power-indices with change points) and estimate the parameters (both change points and indices) - via say MLE - if you like, but in the absence of strong theoretical motivation** that won't mean the result should be given much credence.

** and on the typical "theoretical motivation" for power laws, again see the talk I link above, on which Shalizi is quite dismissive; you can see from the quote I give at the start that he'd be considerably more critical of the notion when it comes to piecewise power laws.

• If you posit a single break, presumably your impulse would be to try to nest the pure power law in the broken (i.e. piece-wise) power law and test the difference, but the problem is that the knot-position is a nuisance parameter that's only there in the alternative, on which see Davies (1987)[1]. (If you know where the changepoint is - say from theory - this isn't an issue.)

Some other things might be done (one might perhaps be able to bootstrap an interval for the change in parameter, or one might try a Bayesian approach, probably via MCMC), but the problem is nontrivial (some people do perform inference on the equivalent problem in regression in a number of ways; it's not an insurmountable problem).

All that said, you might possibly find the discussion in [2] (which relates to changing tail index in time series, so it's not necessarily directly relevant) of some use.

[1] Davies, R.B. (1987),
"Hypothesis testing when a nuisance parameter is present only under the alternative,"
Biometrika 74, 33–43.

[2] Gadeikis, K. and Paulauskas, V. (2005),
"On the Estimation of a Changepoint in a Tail Index,"
Lithuanian Mathematical Journal, 45, 272-283
http://www.mif.vu.lt/katedros/mak/doc/paulauskas/kgvptail3.pdf