Timeline for How to assess repeatability of multivariate and method-specific outcomes?
Current License: CC BY-SA 3.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 18, 2012 at 20:41 | history | edited | Michael R. Chernick | CC BY-SA 3.0 |
deleted 2 characters in body
|
| Jun 17, 2012 at 13:15 | comment | added | cardinal | Downvote removed. | |
| Jun 17, 2012 at 13:14 | comment | added | cardinal | I should have been more specific; by "naive application", I meant that one cannot simply apply the standard test immediately. At the very least some adjustment for degrees of freedom must be made, though sometimes determining what the degrees of freedom should be is not a completely straightforward matter. | |
| Jun 17, 2012 at 12:05 | history | edited | Michael R. Chernick | CC BY-SA 3.0 |
added 592 characters in body
|
| Jun 17, 2012 at 11:51 | comment | added | Michael R. Chernick | However I do not agree with the statement above "A naive application of a χ2 test with data-driven bin selection is invalid". A data driven approach to bin selection for a chi-square test is not invalid. The difficulty is in achieving it and not its validity once achieved. So I will modify or delete my answer. | |
| Jun 17, 2012 at 11:49 | comment | added | Michael R. Chernick | After giving this more thought I think that my recommendation would not work in high dimensions because (1) although a judicious choice of bins is practical in 1, 2 and possibly 3 dimensions, it does not seem to me that identifying such bins in 30 dimensions could be done (2) because of the curse of dimensionality even if such a selection could be achieved points in 30 dimensions spread out in such a way that it would be difficult to detect differences between the distributions without a very large number of points. So cardinal makes some good points. | |
| Jun 17, 2012 at 4:09 | comment | added | cardinal | (-1) It is with considerable reluctance that I downvote, but I don't believe the answer as currently written sufficiently responds to the OP's question. A naive application of a $\chi^2$ test with data-driven bin selection is invalid and there are rather severe difficulties in the first place given the high dimensionality (even the simplest of schemes would require a sample size at least an order of magnitude or two larger than the Earth's human population). I am happy to remove the downvote upon further revision. Cheers. | |
| Jun 17, 2012 at 3:24 | comment | added | Michael R. Chernick | I imagine that there may be a need for a large number of bins but they can be concentrated in the region where the data fall. For the test we just need several high dimensional bins and not 2 for each dimension. | |
| Jun 17, 2012 at 3:16 | history | edited | Michael R. Chernick | CC BY-SA 3.0 |
edited body
|
| Jun 16, 2012 at 20:42 | comment | added | cardinal | @MichaelChernick: Please provide some detail in your answer on how you propose to do this to yield something that would lead to a valid $\chi^2$ test. (What I describe does yield rectangular bins of the simplest form such that no dimension of the data is completely ignored.) | |
| Jun 16, 2012 at 20:35 | comment | added | Michael R. Chernick | @bgbg It is always a judgement call as to how many bins to have and whether or not they should have equal size. I think one tties to keep the bin sizes the same and the number of bins should be chosen to try to reasonable represent the shape of the distribution. Too few bins hides the shape while two many creates too many sparse and empty bins. | |
| Jun 16, 2012 at 20:31 | comment | added | Michael R. Chernick | @cardinal No what I said was to construct 30 dimensional rectangular shaped bins. I do the usual chi-square test for comparing two distributions. | |
| Jun 16, 2012 at 12:49 | comment | added | Boris Gorelik | also, according to your suggestion, it is not clear how the binning should be done: should every bin have the same number of cases, same range, same log range etc? | |
| Jun 15, 2012 at 23:19 | comment | added | cardinal | Hi, Michael. Do you mind clarifying what you're suggesting regarding binning? It sounds like you're suggesting binning each dimension separately and then classifying into bins. But, let's say we have two bins per dimension, that's $2^{30} > 10^9$ bins. That doesn't sound like a good candidate for a $\chi^2$ test. So, what are you suggesting? | |
| Jun 15, 2012 at 21:45 | history | answered | Michael R. Chernick | CC BY-SA 3.0 |