1
$\begingroup$

Let say I observe a phenomenon an I get its PDF. Then I come up with two models to simulate this phenomenon. From these models, I can get a PDF and I want to know which model is better.

I am not interested in the $Q_2$ of the model. Here the objective is to get the PDF to assess threashold exceeding or to see how the phenomenon responds to uncertainties for instance. Maybe there is a bi-modal structure, particular values with particular level of probabilities, etc. This is for exploratory so no particular quantile is targeted and thus I need the PDF from the model which represents best the PDF from the real observations.

So let’s say I have:

  • One sample from observation giving PDF1,
  • One sample from the first model giving PDF2,
  • One sample from the second model giving PDF3.

I am not a statistician, so from my findings I have to use a Kolmogorov-Smirnov test to assess if two PDFs can be considered "equal".

Thus, I compute two-sample Kolmogorov-Smirnov test with PDF1 vs PDF2 giving pvalue1 and from PDF1 vs PDF3 giving pvalue2.

Can I compare the two pvalues? If pvalue1 < pvalue2, can I say that the first model is better?

Or maybe this approach is wrong, so how to tell which model is better at getting the PDF (only the PDF, I know about Q2 and this is not what I am looking for)?

Improve this question
$\endgroup$
25
  • $\begingroup$ What would the two samples be? So far your description implies only one sample: namely, your observations. $\endgroup$
    – whuber ♦
    Commented Jun 8, 2018 at 16:50
  • $\begingroup$ @whuber there is the observation and then using each model you have new predictions. Is that clear? $\endgroup$
    – tupui
    Commented Jun 8, 2018 at 17:39
  • $\begingroup$ Yes, but why are you going about it this way? The one-sample K-S test specifically compares a sample to a distribution: there's no need to generate a new artificial "sample" to run the test. Your basic problem is that when you generate a model from data, it's not valid to test the model against the same data: the logic is circular and, even though you can carry out the K-S calculations, its p-value won't be meaningful. If you're concerned about predictive validity, then the K-S test is practically irrelevant anyway. $\endgroup$
    – whuber ♦
    Commented Jun 8, 2018 at 19:21
  • 1
    $\begingroup$ @Y0da whuber was not being rude, but rather trying to help you solve your problem. You have posted a question in the form of an XY problem (see also wikipedia) - asking about an attempted solution to a problem (simulating and using two sample KS tests) rather than explaining the problem itself. Your attempted approach is clearly unsuitable but you don't explain your underlying situation clearly enough to really offer good advice about what you should do instead.. $\endgroup$
    – Glen_b
    Commented Jun 9, 2018 at 5:29
  • 1
    $\begingroup$ Search first, though, as whuber advised way back near the start of these comments. $\endgroup$
    – Glen_b
    Commented Jun 10, 2018 at 11:56

1 Answer 1

Reset to default
1
$\begingroup$

A two-sample Kolmogorov-Smirnov test comparing data to values simulated from a model is not a good way to see whether a distributional model fits the data.

In the case where you have not estimated any parameters, it's simply a much less efficient way of doing a one-sample K-S test. You can improve its efficiency by drawing a larger simulation sample. As the sample size goes to infinity, guess what you end up with? The one-sample K-S test!

Neither test is suitable (as is) if you estimate parameters, though.

One of the things you're after is to be able to estimate the probability to exceed a threshold. One way to get it is to try to identify a pdf, but it's often not the best way to achieve that goal (in part it depends on how far into the tail the threshold is). If the thresholds are near the middle, you may, for example, be better to estimate these directly from the data.

if those threshholds are not mostly near the middle, it may make sense try to identify a pdf for that purpose, but then I probably wouldn't be using K-S distance to assess fit, but something that would do better in the tail.

(You can also get a good idea how a system behaves without identifying a distributional form for data, since you have the ECDF itself)

However; if you need information about the extreme tail (beyond almost all the data) then the data isn't much use; you need some assumptions as well.

Now lastly, none of this is necessarily of the greatest use for deciding between two competing models.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Some points, but this answer does not answer the last question. If this method is not good, what to use? As I explained I really want the PDF. I want to know which model of my creation will be better at estimating the PDF (and when I have the model, I can create an near to infinite sample). This is for exploratory reason. Threashold exceeding is just an example and prior to getting the PDF, we just don’t know what we want to do with the problem. $\endgroup$
    – tupui
    Commented Jun 10, 2018 at 12:15
  • 1
    $\begingroup$ The last question (which you added yesterday) makes this question overly broad. If you deny any specific purpose and offer no explicit loss function there's not much to say (other than "when you have a loss function, minimize it"). This issue was explained in comments. I have given examples of the sort of considerations that come into designing a loss function for some purpose. If you want to use KS distance as a loss you can do so but (besides it appearing to be arbitrarily chosen), the problem is you cannot reasonably compare models with different numbers of parameters that way. ... ctd $\endgroup$
    – Glen_b
    Commented Jun 11, 2018 at 0:37
  • $\begingroup$ ctd... You might try out of sample loss (via cross validation perhaps); this would make particularly good sense if your problems in the future were mostly predictive in nature $\endgroup$
    – Glen_b
    Commented Jun 11, 2018 at 0:37
  • $\begingroup$ that is the whole point of my question. Is this valid to use KS as a metric. From what you say now the answer is yes although you say this metric would be arbitrary (I just do not understand what are the parameters). But this is true for all metrics, all are arbitrary chosen. In this case I wanted to know if the community had came up with a standard way of doing this or not as I did not find it in the literature. Again, in uncertainty propagation you want the whole PDF and this is the final result. So if you are able to compare your model with a database you want a metric for that. $\endgroup$
    – tupui
    Commented Jun 11, 2018 at 15:57
  • $\begingroup$ No, not all metrics are arbitrary; if they're chosen to suit what you're doing they're not arbitrary. There are a variety of ways people use to choose between competing distribution choices -- many questions on site relate to this. $\endgroup$
    – Glen_b
    Commented Jun 11, 2018 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.