Skip to main content
5 of 5
added 455 characters in body
tupui
  • 41
  • 9

Use results from two-sample Kolmogorov-Smirnov to compare methods

Let say I observe a phenomenon an I get its PDF. Then I come up with two models to simulate this phenomenon. From these models, I can get a PDF and I want to know which model is better.

I am not interested in the $Q_2$ of the model. Here the objective is to get the PDF to assess threashold exceeding or to see how the phenomenon responds to uncertainties for instance. Maybe there is a bi-modal structure, particular values with particular level of probabilities, etc. This is for exploratory so no particular quantile is targeted and thus I need the PDF from the model which represents best the PDF from the real observations.

So let’s say I have:

  • One sample from observation giving PDF1,
  • One sample from the first model giving PDF2,
  • One sample from the second model giving PDF3.

I am not a statistician, so from my findings I have to use a Kolmogorov-Smirnov test to assess if two PDFs can be considered "equal".

Thus, I compute two-sample Kolmogorov-Smirnov test with PDF1 vs PDF2 giving pvalue1 and from PDF1 vs PDF3 giving pvalue2.

Can I compare the two pvalues? If pvalue1 < pvalue2, can I say that the first model is better?

Or maybe this approach is wrong, so how to tell which model is better at getting the PDF (only the PDF, I know about Q2 and this is not what I am looking for)?

tupui
  • 41
  • 9