I want to test if two samples, which follow a skewed distribution, can be distinguished from each other. A Kolmogorov-Smirnov statistic for two samples would be sufficient if there was a way to include errors. Each data point that I have has an error measurement attached to it and these errors are relatively large and vary wildly between each other.
In the following figures, I plot the histograms of two data sets each containing two samples, a red sample and a blue sample. The vertical line denotes the mean. As is visible, in both cases the two sample distributions are skewed to each other. There are around ~240 data points in each histogram for both data sets, where the red and blue sample within a data set are of the same size. The data are restricted in the range [-0.2, 0.4] and the size of the average error for each data point is 0.12 (welcome to my world!).
A Kolmogorov-Smirnov statistic for 2 samples gives me that the p-value is 6e-79 with a KS statistic of 0.85 for the first data set and p=6e-40 with a KS statistic of 0.57 for the second data set (calculated with
scipy.stats.ks_2samp). As you can guess from the size of the average error bar, that sounds too great to be true.
Searching the internet for a weighted Kolmogorov-Smirnov test taking into account errors unfortunately yields nothing. That is imho what I need, unless I can get some statistical reassurance that I do not need to worry on the data point errors in my case.