# Use of two-sample Kolmogorov-Smirnov test to evaluate similarities between two different distributions

$X_1, X_2, \dots X_n$ and $Y_1, Y_2, \dots Y_n; n = 1000$ are two samples of physical quantities coming from the application of two different mathematical models to some independent and identically distributed (iid) data.

The mathematical model used to generate $Y_i%$ is a simplified version (tuned by the parameter $r$) of the mathematical model used to generate $X_i$.

My goal is to find the value of $r$ that makes the empirical distribution $F_Y(y)$ to be as similar as possible to the distribution $F_X(x)$.

I decided to use the Two-Sample Kolmogorov-Smirnov test (in R) for different value of $r$ ranging in a specific interval. Is this choice correct?

I know that the null hypothesis for the K-S test is that the two distributions are the same. However, I know for sure that the two distributions are different because the two mathematical models are different. Is it correct to evaluate the best value of $r$ by looking at the p-value and the D statistic coming from the K-S test?

• Welcome to the site, @RiccardoCavallari. Please don't sign your posts. Notice that your username (with a link to your user page) & your flair are automatically added to your posts. Regarding your question, do you care if the mean or SD is similar, or only if the shape & ranking is similar? Could you just correlate the 2? May 13, 2013 at 14:01
• Thank you @gung, actually, since the two distributions are surely different, it could be enough to find the value of the parameter r that makes the mean and the standard deviation similar. If this is the case, what are the best tools to use? May 13, 2013 at 14:31

You say you're trying to make the two distributions close; the $D$ statistic measures the discrepancy between the two, and (as long as it's not changing the dimension of the fit) choosing $r$ to minimize that discrepancy makes complete sense.

I don't think there's any need to deal with the $p$-value; $D$ is a sensible thing to optimize.

If you're changing the dimension of the fit (adding or removing parameters), just minimizing $D$ isn't going to be sufficient, since more parameters will always tend to improve the fit.

• Thank you! Can you please go into more details about changing the dimension of the fit? May 13, 2013 at 14:19
• It's not clear to me what information you might need. If changing $r$ does something like change the amount of smoothing, you may change the dimension of the model with $r$. Are you able to describe what $r$ does? May 13, 2013 at 14:26
• But if the $p$-value is large, does it matter if $D$ is small w.r.t. establishing similarity? Dec 9, 2013 at 0:31
• That's not necessarily a question I can answer for you, since the relevance will depend on your personal tradeoff between those things. If D is small and p is large, you have no evidence that they're different; if D is large and p is large, it may simply be lack of power. Lack of rejection doesn't imply they're similar; if you want to do that you might want to look into some form of confidence interval. Dec 9, 2013 at 3:41