Timeline for Use results from two-sample Kolmogorov-Smirnov to compare methods
Current License: CC BY-SA 4.0
31 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 10, 2018 at 12:09 | comment | added | Glen_b | It doesn't have an answer though. If you have tried a number of searches but can't find any answers, you can post it. | |
| Jun 10, 2018 at 12:05 | comment | added | tupui | @Glen_b indeed I see this one stats.stackexchange.com/q/25923/140570 | |
| Jun 10, 2018 at 11:56 | comment | added | Glen_b | Search first, though, as whuber advised way back near the start of these comments. | |
| Jun 10, 2018 at 11:55 | answer | added | Glen_b | timeline score: 1 | |
| Jun 10, 2018 at 11:43 | comment | added | tupui | @Glen_b OK I will do this and let you know when the question is out thanks. | |
| Jun 10, 2018 at 11:41 | comment | added | Glen_b | Now that's a good question to ask -- and a reasonable answer is too long for a comment. Why not post that one? You can leave this one if you like; I'll try to turn my comments into an answer here. | |
| Jun 10, 2018 at 11:36 | comment | added | tupui | @Glen_b OK... I really do not get this then. Thanks for your time. Any advice then about the way I should compare my models? | |
| Jun 10, 2018 at 11:34 | comment | added | Glen_b | No, the two-sample version has exactly the same problem (it's wrong if you do parameter estimation), plus it has a number of additional issues. It's worse than doing the one-sample test which is already wrong. I think there's now enough here for the question to be more or less answerable, but most of the issues here are addressed in other questions already on site. | |
| Jun 10, 2018 at 11:32 | comment | added | tupui | @Glen_b That’s my understanding and this is why I wanted to use the two-sample version. So is this correct? | |
| Jun 10, 2018 at 11:30 | comment | added | Glen_b | It's not the one sided version of the KS test that's at issue, it's the one sample version. But you can't use it directly if you're estimating parameters | |
| Jun 10, 2018 at 11:29 | comment | added | tupui | @Glen_b thanks I updated the question. But I do not understand how a can use the one sided KS here. Or maybe I just do not get the real difference between these two tests. | |
| Jun 10, 2018 at 11:27 | history | edited | tupui | CC BY-SA 4.0 |
added 455 characters in body
|
| Jun 10, 2018 at 11:26 | comment | added | Glen_b | ctd... You can also get a good idea how a system behaves without identifying a distributional form for data, since you have the ECDF itself | |
| Jun 10, 2018 at 11:19 | comment | added | Glen_b | It would be a good idea to put your most recent comment in the question. "The probability to exceed a threshold" is certainly a goal. One way to get it is to try to identify a pdf, but it's often not the best way to achieve that goal (in part it depends on how far into the tail the threshold is). If I were nevertheless going to try to identify a pdf for that purpose, I probably wouldn't be using K-S distance but something that would do better in the tail. Then, if you were nevertheless to use KS distance anyway, you don't do it the way you were trying to do it (as whuber explained). ...ctd | |
| Jun 10, 2018 at 7:44 | comment | added | tupui | @Glen_b the goal is the PDF. In risk analysis for instance, they are interested in the probability to exceed a threashold. And sometimes you just do not have any threashold but you are interested to know how the system responds (uncertainty quantification). I am in this framework. I need to have a mean to compare the ability of a surrogate to retrieve the PDF. | |
| Jun 10, 2018 at 7:35 | comment | added | Glen_b | 1. What's the point in estimating a pdf? It's not normally a goal in and of itself -- after you estimate a pdf, you're using that estimated pdf to do something, right? It's that that you should seek to do well at. 2. Closer measured how? With two values there's an infinite number of ways to define a distance between them. But these are two functions. You're simply substituting one undefined term (closer) for another (best). | |
| Jun 10, 2018 at 6:11 | comment | added | tupui | @Glen_b I do not have any more context than comparing two surrogate model’s capabilities to recover a PDF. Best here means that on overall the PDF from one surrogate would be closer than with the other surrogate. So there is no particular focus on some specific quantiles. | |
| Jun 9, 2018 at 23:57 | comment | added | Glen_b | Thanks for your efforts, you're getting closer to an answerable question that has a hope of giving you useful advice, but a little more is needed. "Best" is not defined yet. If you don't have an operational definition of 'best' for your problem (essentially, an algebraic formula or something from which one could be derived) - and I suggest you don't attempt one - you'll require advice about what might at least do fairly well in your circumstances. The problem is we shouldn't be suggesting any if we don't have a good understanding of what you'll be doing with it (i.e. without the circumstances) | |
| Jun 9, 2018 at 6:50 | history | edited | tupui | CC BY-SA 4.0 |
added 2 characters in body
|
| Jun 9, 2018 at 5:52 | history | edited | tupui | CC BY-SA 4.0 |
added 296 characters in body
|
| Jun 9, 2018 at 5:48 | comment | added | tupui | @Glen_b I am not a statistician at all. From my search, I found that KS test would be the way to assess if two PDFs can be considered equal. Maybe this is not the correct approach, but I just don’t know why. As I said in my last comment to whuber, I want to tell which model give me the best PDF. | |
| Jun 9, 2018 at 5:29 | comment | added | Glen_b | @Y0da whuber was not being rude, but rather trying to help you solve your problem. You have posted a question in the form of an XY problem (see also wikipedia) - asking about an attempted solution to a problem (simulating and using two sample KS tests) rather than explaining the problem itself. Your attempted approach is clearly unsuitable but you don't explain your underlying situation clearly enough to really offer good advice about what you should do instead.. | |
| Jun 8, 2018 at 20:58 | comment | added | whuber♦ | I am sorry you find this exchange rude--I'm only trying to assist you in formulating an effective question. Of course I'm not fully understanding, but that's a sign many others who might be in a position to help you might fail to understand or even misunderstand what you're after. You're certainly welcome to keep your question up, but my impression is that in its present form it's not likely to get answers that truly help, and the comments haven't clarified much. | |
| Jun 8, 2018 at 20:47 | comment | added | tupui | @whuber why being so rude!? It is not because I am not using the right wording and that you do not get what I à asking that you can bully. And your reputation here does not give you this right either. So going back to the question. I have two different surrogate models trying to predict some physics and the objective is to compute the PDF. So I do not care here about the predictivity coefficient Q2. So now, how to say that model A is better than model B to get the PDF. Given that I have the budget to get the empirical PDF by doing a large sampling. | |
| Jun 8, 2018 at 20:39 | comment | added | whuber♦ | We seem to be going back and forth. As far as I can tell, you have never described more than one actual sample: everything else comes from some model you have created. There's no need to sample from a model in order to compare it to data. Rather than insisting on this approach, you would be better off abandoning it and changing your question so that it formulates a clear objective, describes the information you have, and asks how to achieve that objective. You will likely discover that our site has a great many threads about model selection and comparison, so consider searching. | |
| Jun 8, 2018 at 20:35 | comment | added | tupui | @whuber I am using the two sided test because I just have samples from the observation. My understanding is that the one sample KS requires a distribution and some sample. Here I have two sets of sample. Hence the two sided test. The goal of all this is to find which model is better at finding the observed PDF. So I thought that comparing pvalues could do this. | |
| Jun 8, 2018 at 19:21 | comment | added | whuber♦ | Yes, but why are you going about it this way? The one-sample K-S test specifically compares a sample to a distribution: there's no need to generate a new artificial "sample" to run the test. Your basic problem is that when you generate a model from data, it's not valid to test the model against the same data: the logic is circular and, even though you can carry out the K-S calculations, its p-value won't be meaningful. If you're concerned about predictive validity, then the K-S test is practically irrelevant anyway. | |
| Jun 8, 2018 at 18:55 | history | edited | tupui | CC BY-SA 4.0 |
added 304 characters in body
|
| Jun 8, 2018 at 17:39 | comment | added | tupui | @whuber there is the observation and then using each model you have new predictions. Is that clear? | |
| Jun 8, 2018 at 16:50 | comment | added | whuber♦ | What would the two samples be? So far your description implies only one sample: namely, your observations. | |
| Jun 8, 2018 at 14:29 | history | asked | tupui | CC BY-SA 4.0 |