# What distribution should I fit to this data

I am trying to find a distribution that fits my data (3500+ data points) with satisfying goodness of fit (gof), I use the Kolmogorov-Smirnov test and its p-value as a gof measurement (p-value > 0.1).

I have tried the plfit.m and plpva.m program from Clauset et al. to fit a power-law distribution to my data, but the p-value is close to zero indicating it is not a good fit.

The log-normal and exponential distribution is also tested using the R package poweRlaw, but I still can't get a good enough p-value (> 0.1). However, I think the fitted curve is just close enough to the empirical data, as the picture shows (generated by the poweRlaw package).

I am completely new to this kind of job but I have to report the gof in my paper, so I am wondering:

1. Did I do the fitting job in the correct way? (I didn't modify the Matlab or R programs)
2. Is Kolmogorov-Smirnov test a proper approach to measure the gof?
3. If I remove the extreme cases (2 data points on the right tail), will it become better?
4. What distribution should I fit my data?

Thanks!

• Why are you trying to find a distribution that fits your data? How was the data created? What was measured? Dec 13 '19 at 7:50
• @Roland I constructed diffusion networks of ideas, in which the nodes and edges represent publications and citations. Each idea has a diffusion network, this data represents the distribution of the edge count of a network. The x- and y-axis represent the edge count and its corresponding cumulative probability. I want to describe some characteristics of it, so I think I should find it a proper distribution and describe the characteristics of that distribution. Dec 13 '19 at 10:37
• The link points to data with only one column (the x-values, most likely). Could you provide the y-values, too? Would make testing much easier... Dec 14 '19 at 18:14
• @PatrickHappel I have updated my data with y-values, I am completely lost to find an appropriate distribution. Maybe the power-law distribution with xmin and xmax would help? But I don't know how to perform k-s test to this kind of truncated distribution. Is there any way to validate the gof? I really need some help. :( Dec 15 '19 at 2:27

I've plotted log(data) vs abscissa and dropped two last points, graph below

data going below x=500 would be good fitted with just a linear function - which means exponential distribution because we plotted log(data).

But from 0 to 500 things looks different, maybe power law? Thus, I don't know if data could be fitted with one distribution function.

UPDATE

I played with truncated power law, and indeed this might work

Code (Python3.7, Anaconda x64 Win 10)

import numpy as np
from scipy.special import gamma
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

xs = []
ys = []

with open('my_data.csv') as f:
for line in f:
x, y = line.split(',')
xs.append(float(x))
ys.append(float(y))

xs = np.asarray(xs)
ys = np.asarray(np.log(ys))

x0 = xs[0]

def F(x, a, b):
return np.power((x-x0)+1.0, -a) * np.exp(-b*(x-x0))

def logF(x, a, b):
return -a*np.log((x-x0)+1.0) + (-b*(x-x0))

popt, pcov = curve_fit(logF, xs, ys)
print(popt)

plt.plot(xs, ys, 'b*', label='data')
plt.plot(xs, logF(xs, *popt), 'g-', label='fit')
plt.show()


produced reasonable fit graph. And if you print sqrt of pcov diagonal, errors looks small

print(popt)
print(np.sqrt(np.diag(pcov)))


produced

[0.38962489 0.00140291]
[2.51804031e-03 2.10496530e-05]


• Thanks a lot! If I only fit the partial data with a distribution, can I still perform a K-S test on it? If so, how? Dec 17 '19 at 5:12
• The python powerlaw package provides a truncated power-law fit (exponential cutoff) and I think it's better, but I don't know how to perform K-S test on it. Also, I can't figure out what is the CDF of a power-law with an exponential cutoff. Dec 17 '19 at 5:24
• @user99625 please check update, fitted values for [a, b] are [0.38962489 0.00140291] Dec 17 '19 at 16:02
• @user99625 Frankly, is you took out data shift, it looks at the end as Gamma distribution Dec 17 '19 at 17:38
• @user99625 concerning K-S test, if you could by shifting, scaling and normalization make your fit into Gamma distribution, then K-S test should work with easy - CDF of Gamma distribution is well-known. Dec 18 '19 at 21:14