I have a dataset that contains the distance (in cm) between the emission point and the interaction point of about 50 000 fluorescence X-rays. When I plot its histogram, I should expect an exponential probability density function. In Python, I would like to recover the parameters of this pdf using
scipy.stats.expon.fit() and test the goodness of this fit using a Kolmogorov-Smirnov test (with
scipy.stats.expon.kstest()). However, everytime I try to test my fit, a get a p-value of
0.0, which seems to indicate that the null hypothesis should be rejected (ie. that my data doesn't follow an exponential pdf), but when I plot both the histogram and the fitted pdf, it seems alright.
Here are the details.
As you can see, the histogram doesn't go to 0 because of the resolution of my pixelated detector (it won't recognize the emission and the absorption of the X-ray if they happens in adjacent pixels, which are about 55*55 micrometers). However, it shouldn't be a problem since the fit also returns the
loc parameter and because I'm mostly interested in the
scale parameter (to get the mean free path of the X-rays).
Also, I have a lot of noise, and it especially affects the tail of the distribution. So I don't really want the fit to be precise for higher values, really what I care about is the decreasing rate of the distribution for small values.
So let's say that I loaded my 50 000 values in an array called
data. Here's what I tried:
>>> import numpy as np >>> import scipy.stats as scp >>> loc,scale=scp.expon.fit(data[np.where(data<0.1)]) #Fit the left part of the distrib. >>> scp.kstest(data[np.where(data<0.1)],"expon",args=(loc,scale)) #KS Test (0.11032993451965302, 0.0)
So here's the
0.0 p-value. However, if I compare the fit and the dataset graphically, it doesn't seem so bad:
>>> import matplotlib.pyplot as plt >>> plt.hist(data[np.where(data<0.1)], bins=150, normed=True, histtype="step") >>> x=np.linspace(0,0.1,200) >>> y=scp.expon.pdf(x,loc,scale) >>> plt.plot (x,y) >>> plt.show()
I think that the problem is that I try to fit only part of the probability density function and that the KS test checks for the presence of the tail. Any solution? Is there another test I should use? I'm no statistician, I've never really made a fit for a continuous pdf before, I don't know if KS test is the best choice or if I use it correctly. Thanks!