2
$\begingroup$

It seems like a proposed study gets penalised if it doesn't also predict the direction of an effect. For instance, if I want to see the effect of a certain medication on happiness ratings, but do not predict whether I think they'll be decreasd or increased by that manipulation. Why is it considered unsound to run an exploratory study, i.e. one that doesn’t have a specific prediction (hypothesis) but simply wants to see what effect a certain manipulation has on another dependent variable (i.e. you have a hunch there's an effect there but don’t quite know in which direction it'll be)?

In other words, what exactly is the formal statistical principle that is violated if you run a study without formulating an apriori hypothesis? It seems that if you're happy to pay the 'penalties', e.g. use 2-tailed instead of 1-tailed tests and correct for all the multitude of tests, any exploratory result should be just as believable as those of a hypothesis-driven study!

Improve this question
$\endgroup$
7
  • $\begingroup$ Use of a 2-tailed test when the direction of an effect isn't pre-specified is "penalization" enough. What "multitude of tests" are you thinking about? $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 25, 2014 at 19:04
  • $\begingroup$ Sorry, I was unclear, I was referring to another possible exploratory study whereby you're just looking for correlations between various variables any pair of which you'd expect could be correlated $\endgroup$
    – z8080
    Mar 25, 2014 at 19:09
  • 2
    $\begingroup$ Yes (you should perhaps edit the question rather than adding clarification or extra detail in comments). I haven't answered simply because I haven't come across such attitudes (perhaps you could provide some references). What is widely denigrated (whether or not it's considered worthwhile in a particular case to correct p-values or widen confidence intervals to account for multiple comparisons), is deciding the form of the analysis based on the results of the experiment - transforming the response, re-grouping categorical predictors, choosing a test statistic, &c.; generally tweaking ... $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 30, 2014 at 13:30
  • 2
    $\begingroup$ ... things to give the "best" results - & then keeping mum about all of that; disregarding these "researcher degrees of freedom" in the final analysis. For alleged examples see Bem (2011), "Feeling the Future", JPSS, 100, 3 & Witztum et al. (1994), "Equidistant Letter Sequences in the Book of Genesis", Stat. Sci., 9, 3. Every analysis involves some unknowns, otherwise there'd be no point to it, but you should try to make ... $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 30, 2014 at 13:31
  • 2
    $\begingroup$ ... the formal analysis broad enough to be able to treat them formally. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 30, 2014 at 13:34

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.