I want to show the confidence envelope for a two sample Q-Q plot in R (or Python). The aim is to use the Q-Q plot to give an indication of whether my two samples are drawn from the same population
The method of qqPlot() from the car package cannot do this, because this method does not support two sample tests, I think. The method of qqplot() from the base R installation cannot do this by itself, since it does not support confidence intervals, but it does support two samples.
Is it possible to use Kolmogorov-Smirnov or bootstrapping for this?
Thank you for your time!
Edit: I am hijacking my own question.
Part of the background for why I am asking is because I feel the Kolmogorov-Smirnov test (KS) and Anderson-Darling test (AD) give me misleading results for my comparisons. In my research I am interested in answering whether a factor might influence the size distributions of particles in my material. An example of KDEs, CDFs, Q-Q plot, and KS and AD test results for graphite particles are provided below (everything is calculated and plotted in Python):
KS p-value: 6.58*10^-6.
AD p-value: 0.001.
Number of particles counted: approximately 1000. (I know that for huge sample sizes [~5000] goodness of fit tests such as KS tend to give very low p values even if the deviation from normality is quite low (for one sample tests)).
Q-Q plot of samples, made in SciPy by first obtaining the quantiles with the statsmodels.graphics.gofplots.ProbPlot function, then plotting them using statsmodels.graphics.gofplots.qqplot_2samples:
I guess my second question is: Is my Q-Q plot and KS test result in conflict with each other? Should I focus on the Q-Q plot and disregard KS?
Edit: The culprit for the apparent conflicting results was a mistake when plotting the Q-Q plot. The Q-Q plot matches the KS results now. Thank you for the nice suggestion regarding permutations!