5
$\begingroup$

I am trying to find a distribution that fits my data (3500+ data points) with satisfying goodness of fit (gof), I use the Kolmogorov-Smirnov test and its p-value as a gof measurement (p-value > 0.1).

I have tried the plfit.m and plpva.m program from Clauset et al. to fit a power-law distribution to my data, but the p-value is close to zero indicating it is not a good fit.

The log-normal and exponential distribution is also tested using the R package poweRlaw, but I still can't get a good enough p-value (> 0.1). However, I think the fitted curve is just close enough to the empirical data, as the picture shows (generated by the poweRlaw package).

I am completely new to this kind of job but I have to report the gof in my paper, so I am wondering:

  1. Did I do the fitting job in the correct way? (I didn't modify the Matlab or R programs)
  2. Is Kolmogorov-Smirnov test a proper approach to measure the gof?
  3. If I remove the extreme cases (2 data points on the right tail), will it become better?
  4. What distribution should I fit my data?

Thanks! enter image description here

$\endgroup$
4
  • $\begingroup$ Why are you trying to find a distribution that fits your data? How was the data created? What was measured? $\endgroup$
    – Roland
    Dec 13 '19 at 7:50
  • $\begingroup$ @Roland I constructed diffusion networks of ideas, in which the nodes and edges represent publications and citations. Each idea has a diffusion network, this data represents the distribution of the edge count of a network. The x- and y-axis represent the edge count and its corresponding cumulative probability. I want to describe some characteristics of it, so I think I should find it a proper distribution and describe the characteristics of that distribution. $\endgroup$
    – Tom Leung
    Dec 13 '19 at 10:37
  • $\begingroup$ The link points to data with only one column (the x-values, most likely). Could you provide the y-values, too? Would make testing much easier... $\endgroup$ Dec 14 '19 at 18:14
  • $\begingroup$ @PatrickHappel I have updated my data with y-values, I am completely lost to find an appropriate distribution. Maybe the power-law distribution with xmin and xmax would help? But I don't know how to perform k-s test to this kind of truncated distribution. Is there any way to validate the gof? I really need some help. :( $\endgroup$
    – Tom Leung
    Dec 15 '19 at 2:27
1
$\begingroup$

I've plotted log(data) vs abscissa and dropped two last points, graph below

enter image description here

data going below x=500 would be good fitted with just a linear function - which means exponential distribution because we plotted log(data).

But from 0 to 500 things looks different, maybe power law? Thus, I don't know if data could be fitted with one distribution function.

UPDATE

I played with truncated power law, and indeed this might work

Code (Python3.7, Anaconda x64 Win 10)

import numpy as np
from scipy.special import gamma
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

xs = []
ys = []

with open('my_data.csv') as f:
     for line in f:
         x, y = line.split(',')
         xs.append(float(x))
         ys.append(float(y))

xs = np.asarray(xs)
ys = np.asarray(np.log(ys))

x0 = xs[0]

def F(x, a, b):
    return np.power((x-x0)+1.0, -a) * np.exp(-b*(x-x0))

def logF(x, a, b):
    return -a*np.log((x-x0)+1.0) + (-b*(x-x0))

popt, pcov = curve_fit(logF, xs, ys)
print(popt)

plt.plot(xs, ys, 'b*', label='data')
plt.plot(xs, logF(xs, *popt), 'g-', label='fit')
plt.show()

produced reasonable fit graph. And if you print sqrt of pcov diagonal, errors looks small

print(popt)
print(np.sqrt(np.diag(pcov)))

produced

[0.38962489 0.00140291]
[2.51804031e-03 2.10496530e-05]

enter image description here

$\endgroup$
5
  • $\begingroup$ Thanks a lot! If I only fit the partial data with a distribution, can I still perform a K-S test on it? If so, how? $\endgroup$
    – Tom Leung
    Dec 17 '19 at 5:12
  • $\begingroup$ The python powerlaw package provides a truncated power-law fit (exponential cutoff) and I think it's better, but I don't know how to perform K-S test on it. Also, I can't figure out what is the CDF of a power-law with an exponential cutoff. $\endgroup$
    – Tom Leung
    Dec 17 '19 at 5:24
  • $\begingroup$ @user99625 please check update, fitted values for [a, b] are [0.38962489 0.00140291] $\endgroup$ Dec 17 '19 at 16:02
  • $\begingroup$ @user99625 Frankly, is you took out data shift, it looks at the end as Gamma distribution $\endgroup$ Dec 17 '19 at 17:38
  • $\begingroup$ @user99625 concerning K-S test, if you could by shifting, scaling and normalization make your fit into Gamma distribution, then K-S test should work with easy - CDF of Gamma distribution is well-known. $\endgroup$ Dec 18 '19 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.