Timeline for What does "Scientists rise up against statistical significance" mean? (Comment in Nature)
Current License: CC BY-SA 4.0
24 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 30, 2019 at 15:45 | comment | added | Alexis | Well, no, because inference also happens without models (e.g., dichotomous outcomes in randomized control trials), and equivalence tests are also apt there. But sure I will bite, and this is my last comment in this thread: statistical inference is only done in the context of specific models (or designs sans models), and the moment you look for statistical evidence of effect in such a model or circumstance, the moment you commit to a statistical relationship you can and should look for evidence of effect, and evidence of no effect. Period. | |
| Mar 30, 2019 at 7:37 | comment | added | Nat | @Alexis Then I think that this is where we'd disagreed earlier - and, I'm hoping, due to ambiguity in communication. This is, I don't believe that equivalence testing can be used in conjunction with a study between GPA and income to broadly establish that the two are unrelated, or that knowledge of a student's GPA can't be used to inform predictions of their later income -- rather, I see equivalence testing as only being able to make such claims within the context of some specific model, which I don't find to be a generally interesting exercise. Would you agree/disagree? | |
| Mar 30, 2019 at 7:35 | comment | added | Nat | @Alexis The relevance to the article being that, in science, we're not typically interested in establishing effects only within a limited subset, but rather we're greedily searching for any sort of effect that we can describe. For example, one study may fail to find a correlation between income and GPA in one context, but this doesn't generally establish that no such correlation that we might appreciate exists, merely that the specific correlations tested for within the experimental context failed to identify such a correlation. The concern being that claims of unrelatedness don't follow. | |
| Mar 30, 2019 at 7:28 | comment | added | Nat | @Alexis For example, I'll agree that we can establish that, within the context of a linear correlation between $x$ and $y ,$ knowledge of $x$ can't be used to inform predictions of a corresponding $y ,$ which one might then describe as there not being a relationship between the two - which, I assume, is what you mean by using equivalence testing. However, my point's that such an observation would be limited to the scope of the model used to perform the equivalence testing, e.g. the linear correlation; that this can't be used to broadly preclude any potentially interesting correlation. | |
| Mar 30, 2019 at 7:25 | comment | added | Nat | @Alexis As for equivalence testing, I appreciate that it's possible due to a limited number of models. This is, while we can't walk over the general space of all potentially interesting correlations, we can walk over the space of, say, linear correlations. We can then say that the relatively likelihood of a subspace is greater than some threshold of significance, which we can then use as a basis for arguing that the parameters that describe that subspace are valid. Then, for example, we can say that in a linear correlation, the slope is $0 .$ [...] | |
| Mar 30, 2019 at 7:20 | comment | added | Nat | @Alexis Thanks for bearing with me on this. Okay, so, given that we agree that there're an infinity of relationships that can relate any two data series, it's my general position that it's not fruitful to focus on broadly declaring any two data series to be "unrelated", since such a statement would require finding the non-significance of infinitely many correlations that could exist between them. Instead, I see statistics as a constructive tool, where we must focus on finding correlations that tend to work. [...] | |
| Mar 29, 2019 at 15:30 | comment | added | Alexis | Name your statistical model: you can and should look combine evidence for equivalence with evidence for difference. | |
| Mar 29, 2019 at 15:23 | comment | added | Alexis | Nat, I teach nonlinear and nonparametric regression to my grad students, and have some of them read the Nature article by Reshef which uses figures bearing a striking similarity to the uncited Wikimedia image in your answer. So I truly appreciate the issue of infinite possible functional relationships for a continuous IV. That is a separate issue from the one I am raising, which I believe is strongly implied in my second comment in that function form is part of a well-specified causal model. Causal inference $\ne$ statistical inference. | |
| Mar 29, 2019 at 2:01 | comment | added | Nat | I can update the above answer if we can find something that makes sense. I mean, I can appreciate how someone might hear about equivalence testing and think that it can generally detect for the presence-or-absence of a relationship. So, it'd be nice to expand the above answer with an explanation of why equivalence testing can't be used for that purpose, as I can imagine it being a common misunderstanding. | |
| Mar 29, 2019 at 1:54 | comment | added | Nat | @Alexis My third attempt is: Any two data sets, say $x_1$ and $x_2 ,$ are necessarily describable as being related by an infinity of functions, e.g. high-period sine functions and piecewise functions, such that we're always able to identify infinitely many functions that relate any two data series. If you don't constrain your focus to some specific model, then how can you ever say that any two data series aren't related? (Note that, even if you attempt to use validation, the appended data series will still be related by an infinitely large subset of the original infinite set.) | |
| Mar 29, 2019 at 1:47 | comment | added | Nat | @Alexis My second attempt is: Say that I generate a set of data, $f(x),$ as a function of inputs $x \in \left[0, 10\right].$ I'll further guarantee that there's an isometric reverse-mapping, $f^{-1}(x) ,$ such that $x=f^{-1}(f(x)).$ Then, could you use equivalence testing to identify that $x$ and the generated data set are related by a function $f(x) ?$ If so, then that'd be a universal distinguishing algorithm (and be more than worthy of a Fields Medal). If not, then how can you argue that equivalence testing can test for the presence of any relationship? | |
| Mar 29, 2019 at 1:40 | comment | added | Nat | @Alexis I've been trying to think of a good way to communicate the issue, so if you'd indulge me, I'd like to try here. So I guess my first try is this: If you can detect effects up to a certain size without a model, then how do you define "size" without a measure? | |
| Mar 24, 2019 at 7:09 | comment | added | Alexis | I claim nothing of the sort. However you are privileging statistical evidence of difference over evidence of equivalence. That is prima facie confirmation bias, hence my downvote. | |
| Mar 24, 2019 at 6:22 | comment | added | Nat | @Alexis So you're claiming to have a universal distinguishing algorithm? | |
| Mar 24, 2019 at 6:16 | comment | added | Alexis | I am absolutely not making that assumption. You are simply falsely asserting that lack of knowing the true model prevents statistical evidence of equivalence, but somehow does not prevent statistical evidence of difference. If statistics can provide evidence, it can do so for both presence of an effect and absence of an effect larger than a given effect size | |
| Mar 24, 2019 at 5:44 | comment | added | Nat | @Alexis Statistical inference can provide you as much evidence of the absence of an effect larger than a specific effect size within the context of some model. Perhaps you're assuming that the model will always be known? | |
| Mar 24, 2019 at 5:32 | comment | added | Alexis | @Nat You are conflating statistical evidence with causal model. In the world of statistical inference you can provide as much evidence of absence of an effect larger than a specific effect size as you can provide evidence of an effect of a specific effect size. Of course there are causal biases (predicted on specific kinds of relationships) that can moot statistical inferences, but the former apply equally to evidence of effect and evidence of absence of effect. I agree with your tl;dr only in as much as statistics and science never prove: they only give evidence. | |
| Mar 24, 2019 at 5:21 | comment | added | Nat | @Alexis I think you misunderstand equivalence testing; you can use equivalence testing to evidence the absence of a certain relationship holding, e.g. a linear relationship, but not evidence the absence of any relationship. | |
| Mar 23, 2019 at 21:46 | comment | added | Alexis | -1 "tl;dr- It's fundamentally impossible to prove that things are unrelated": Equivalence tests provide evidence of absence of an effect within an arbitrary effect size. | |
| Mar 23, 2019 at 1:55 | comment | added | Nat | @ruakh While I appreciate what you're trying to say, in the above, I was referring to the encryption/decryption transforms. I mean, you're right -- we do all sorts of stuff, like add/remove padding, compress/decompress messages, prepend/remove a nonce, add/check a message's signature with HMAC, add meta-data like a datestamp, etc.. But in the above, I'm specifically talking about the part of crypto that acts on the prepared message to produce the ciphertext and vice-versa, rather than any of the other steps that tend to come before and after that. | |
| Mar 22, 2019 at 23:38 | comment | added | ruakh | This may not be so important for a statistics site, but FYI, most modern encryption schemes do not have the property that "the ciphertext and its corresponding plaintext 100% determine each other". Rather, the encryption process usually incorporates additional random padding; if you encrypt the same message multiple times, you'll get different results each time. (The only exceptions I'm aware of are schemes that aim to have the ciphertext be exactly the same length as the plaintext, e.g. for API compatibility reasons, and are willing to sacrifice a bit of security for that.) | |
| Mar 22, 2019 at 8:05 | comment | added | Nat | @ntg Yeah, it's hard to know how to word some of this stuff, so I left a lot out. I mean, the general truth is that we can't disprove that some relationship exists, though we can generally demonstrate that a specific relationship doesn't exist. Sorta like, we can't establish that two data series are unrelated, but we can establish that they don't appear to be reliably related by a simple linear function. | |
| Mar 22, 2019 at 5:21 | comment | added | ntg | +1 cause what you write is both true and thought provoking. However, in my humble opinion, you can prove that two quantities are reasonably uncorrelated under certain assumptions. You have to offcourse first start by e.g. supposing a certain distribution about them, but this can be based on the laws of physics, or statistics (e.g. the speed of molecules of a gas in a container are expected to be gaussian or so on) | |
| Mar 22, 2019 at 4:21 | history | answered | Nat | CC BY-SA 4.0 |