$X_1, X_2, \dots X_n$ and $Y_1, Y_2, \dots Y_n; n = 1000$ are two samples of physical quantities coming from the application of two different mathematical models to some independent and identically distributed (iid) data.

The mathematical model used to generate $Y_i%$ is a simplified version (tuned by the parameter $r$) of the mathematical model used to generate $X_i$.

My goal is to find the value of $r$ that makes the empirical distribution $F_Y(y)$ to be as similar as possible to the distribution $F_X(x)$.

I decided to use the Two-Sample Kolmogorov-Smirnov test (in R) for different value of $r$ ranging in a specific interval. Is this choice correct?

I know that the null hypothesis for the K-S test is that the two distributions are the same. However, I know for sure that the two distributions are different because the two mathematical models are different. Is it correct to evaluate the best value of $r$ by looking at the p-value and the D statistic coming from the K-S test?

  • 1
    $\begingroup$ Welcome to the site, @RiccardoCavallari. Please don't sign your posts. Notice that your username (with a link to your user page) & your flair are automatically added to your posts. Regarding your question, do you care if the mean or SD is similar, or only if the shape & ranking is similar? Could you just correlate the 2? $\endgroup$ May 13 '13 at 14:01
  • $\begingroup$ Thank you @gung, actually, since the two distributions are surely different, it could be enough to find the value of the parameter r that makes the mean and the standard deviation similar. If this is the case, what are the best tools to use? $\endgroup$
    – Šatov
    May 13 '13 at 14:31

You say you're trying to make the two distributions close; the $D$ statistic measures the discrepancy between the two, and (as long as it's not changing the dimension of the fit) choosing $r$ to minimize that discrepancy makes complete sense.

I don't think there's any need to deal with the $p$-value; $D$ is a sensible thing to optimize.

If you're changing the dimension of the fit (adding or removing parameters), just minimizing $D$ isn't going to be sufficient, since more parameters will always tend to improve the fit.

  • $\begingroup$ Thank you! Can you please go into more details about changing the dimension of the fit? $\endgroup$
    – Šatov
    May 13 '13 at 14:19
  • $\begingroup$ It's not clear to me what information you might need. If changing $r$ does something like change the amount of smoothing, you may change the dimension of the model with $r$. Are you able to describe what $r$ does? $\endgroup$
    – Glen_b
    May 13 '13 at 14:26
  • $\begingroup$ But if the $p$-value is large, does it matter if $D$ is small w.r.t. establishing similarity? $\endgroup$
    – Jacob
    Dec 9 '13 at 0:31
  • $\begingroup$ That's not necessarily a question I can answer for you, since the relevance will depend on your personal tradeoff between those things. If D is small and p is large, you have no evidence that they're different; if D is large and p is large, it may simply be lack of power. Lack of rejection doesn't imply they're similar; if you want to do that you might want to look into some form of confidence interval. $\endgroup$
    – Glen_b
    Dec 9 '13 at 3:41

This site is temporarily in read only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .