4
$\begingroup$

$X_1, X_2, \dots X_n$ and $Y_1, Y_2, \dots Y_n; n = 1000$ are two samples of physical quantities coming from the application of two different mathematical models to some independent and identically distributed (iid) data.

The mathematical model used to generate $Y_i%$ is a simplified version (tuned by the parameter $r$) of the mathematical model used to generate $X_i$.

My goal is to find the value of $r$ that makes the empirical distribution $F_Y(y)$ to be as similar as possible to the distribution $F_X(x)$.

I decided to use the Two-Sample Kolmogorov-Smirnov test (in R) for different value of $r$ ranging in a specific interval. Is this choice correct?

I know that the null hypothesis for the K-S test is that the two distributions are the same. However, I know for sure that the two distributions are different because the two mathematical models are different. Is it correct to evaluate the best value of $r$ by looking at the p-value and the D statistic coming from the K-S test?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Welcome to the site, @RiccardoCavallari. Please don't sign your posts. Notice that your username (with a link to your user page) & your flair are automatically added to your posts. Regarding your question, do you care if the mean or SD is similar, or only if the shape & ranking is similar? Could you just correlate the 2? $\endgroup$
    – gung - Reinstate Monica
    May 13, 2013 at 14:01
  • $\begingroup$ Thank you @gung, actually, since the two distributions are surely different, it could be enough to find the value of the parameter r that makes the mean and the standard deviation similar. If this is the case, what are the best tools to use? $\endgroup$
    – Šatov
    May 13, 2013 at 14:31

1 Answer 1

Reset to default
5
$\begingroup$

You say you're trying to make the two distributions close; the $D$ statistic measures the discrepancy between the two, and (as long as it's not changing the dimension of the fit) choosing $r$ to minimize that discrepancy makes complete sense.

I don't think there's any need to deal with the $p$-value; $D$ is a sensible thing to optimize.

If you're changing the dimension of the fit (adding or removing parameters), just minimizing $D$ isn't going to be sufficient, since more parameters will always tend to improve the fit.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you! Can you please go into more details about changing the dimension of the fit? $\endgroup$
    – Šatov
    May 13, 2013 at 14:19
  • $\begingroup$ It's not clear to me what information you might need. If changing $r$ does something like change the amount of smoothing, you may change the dimension of the model with $r$. Are you able to describe what $r$ does? $\endgroup$
    – Glen_b
    May 13, 2013 at 14:26
  • $\begingroup$ But if the $p$-value is large, does it matter if $D$ is small w.r.t. establishing similarity? $\endgroup$
    – Jacob
    Dec 9, 2013 at 0:31
  • $\begingroup$ That's not necessarily a question I can answer for you, since the relevance will depend on your personal tradeoff between those things. If D is small and p is large, you have no evidence that they're different; if D is large and p is large, it may simply be lack of power. Lack of rejection doesn't imply they're similar; if you want to do that you might want to look into some form of confidence interval. $\endgroup$
    – Glen_b
    Dec 9, 2013 at 3:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.