12
$\begingroup$

Imagine a researcher is exploring a dataset and runs 1000 different regressions and he finds one interesting relationship among them.

Now imagine another researcher with the same data runs just 1 regression, and it turns out it's the same one that the other researcher took 1000 regressions to find. Researcher 2 does not know researcher 1.

Should researcher 1 make different inferences than researcher 2? Why? For example, should researcher 1 perform multiple comparisons correction, but researcher 2 should not?

If researcher 2 showed you his single regression first, what inferences would you make? If after that researcher 1 showed you his results, should you change your inference? If so, why should it matter?

PS 1: If talking about hypothetical researchers makes the problem abstract, think about this: imagine you ran just one regression for your paper, using the best method available. Then another researcher explored 1000 different regressions with the same data, until he found the exact same regression you ran. Should you two make different inferences? Is the evidence the same for both cases or not? Should you change your inference if you knew the other researcher results? How should the public assess the evidence of the two studies?

PS 2: please try to be specific and to provide a mathematical/theoretical justification, if possible!

Improve this question
$\endgroup$
17
  • 1
    $\begingroup$ To be a valid comparison you need to specify all the null and alternative hypotheses. Researcher 2 can only test 1 hypothesis while Researcher 1 probably wants to control thee probability of not make 1 type 1 error out of 1000. If that is the simultaneous inference that you want to make then you have to make do the p-value adjustment. Researcher 2 has one test and no need for adjustment. For researcher 1 are you fitting different models to the same data or one model fit for each of 1000 data sets? $\endgroup$ – Michael R. Chernick Dec 14 '17 at 7:39
  • 1
    $\begingroup$ @MichaelChernick there's only one dataset. Researcher 1 fits 1000 models for the same dataset until he finds the one he likes. Researcher 2 fitted only 1. Both researchers use the same data. So would you say these two researchers would have to conclude different things with the exact same dataset? Researcher 2 should be rightfully confident of his analysis, while researcher 1 should inflate his p-value/confidence intervals due to multiple comparisons? $\endgroup$ – statslearner Dec 14 '17 at 7:48
  • $\begingroup$ If you followed my argument they do in the sense that only researcher 2 is testing a single hypothesis while researcher 1 is testing 1000 hypotheses and needs to control for all the hypotheses he tested.. It involves two different problems. What is still vague is what you mean by "finding only one interesting relationship". Maybe you think you have posed a paradoxical situation. I don't think that you have. $\endgroup$ – Michael R. Chernick Dec 14 '17 at 8:22
  • 1
    $\begingroup$ @MichaelChernick how is it not a paradox for the exact same data with the exact same model lead to two different conclusions? If you read the two separate papers, what would you conclude? $\endgroup$ – statslearner Dec 14 '17 at 15:22
  • 1
    $\begingroup$ @MichaelChernick I did, and I find troubling you think this is correct --- the exact same data, with the exact same model, leading to two different conclusions. See my comments on the answer. $\endgroup$ – statslearner Dec 14 '17 at 15:39
3
$\begingroup$

Here's my "Bayesian" slant on your question. I think you have described a situation where two people with different prior information should get a different answer/conclusion when given the same dataset. A more blunt/extreme example is suppose that we have a "researcher 1b" who just happens to guess the regression model parameters and conclusions from whatever hypothesis. Running $1000$ regressions is not conceptually too far away from guessing.

What I think is happening...what do we learn about the researchers prior information from the above question? - researcher 1 probably has a flat prior for the models $P (M_k|I_1)=\frac {1}{1000} $ - researcher 2 has a sharp prior for the model of interest $P (M_1|I_2) =1$ (assume $M_1$ is the model they both fit)

This is obviously a simplification, but you can see here, we already place a lot more weight on researcher 2's inferences without any data. But you see, once they both take account of the data, researcher 1's posterior probability for $M_1$ will increase... $P (M_1|DI)>>P (M_1|I) $ (...we know this because it was "better" than $999$ other models...). Researcher 2's posterior can't concentrate anymore, it is already equal to $1$. What we don't know is how much the data supported $M_1$ over the alternatives. What we also don't know is how the different models alter the substantive conclusions of researcher 1. For example, suppose all $1000$ models contain a common term, and all $1000$ regression parameters for that variable are significantly greater than $0$ (eg $p-value <10^{-8}$ for all models). Then there is no problem with concluding a significantly positive effect, even though many models were fit.

You also don't say how big the dataset is, and this matters! If you're talking about a dataset with $100$ observations and $10$ covariates/predictors/independent variables, then researcher 1 will probably still be quite uncertain about the model. However, if researcher 1 is using $2,000,000$ observations, this may conclusively determine the model.

There is nothing fundamentally wrong with two people that start with different information, and continue to have different conclusions after seeing the same data. However...seeing the same data will bring them closer together, provided their "model space" overlaps and the data supports this "overlapping region".

Improve this answer
$\endgroup$
7
  • $\begingroup$ So the fundamental part of your claim is that they should make different inference because they have different priors, and not because how much they "explored the data", correct? $\endgroup$ – statslearner Dec 16 '17 at 6:29
  • $\begingroup$ By the way, how would you assess the evidence? Would you care about how many models researcher 1 fitted? If so, why? $\endgroup$ – statslearner Dec 16 '17 at 6:38
  • $\begingroup$ I wouldn't necessarily care about the number of models fit, but whether or not the model being used is known with a degree of certainty. As I mention briefly, I would want to know if there were reasonable alternatives. For example, if researcher 1 made a "line ball" decision about a variable being dropped/added, I would want to see that mentioned. $\endgroup$ – probabilityislogic Dec 16 '17 at 11:13
  • 1
    $\begingroup$ Why would you want to see that mentioned, would that change your prior somehow? Are you using him as proxy for you prior? It's not clear to me the mapping you are making. Why do the reasons of a particular researcher matter for your inference, since it does not affect the data generating process at all? $\endgroup$ – statslearner Dec 16 '17 at 16:03
  • 1
    $\begingroup$ We are considering the dataset here to be external to the researcher, he did not collect it, and both researchers use the same data. It seems the reasons findings cannot be replicated in psychology is because they just use loose significance thresholds as the standard of evidence to judge several crazy hypothesis any reasonable person/scientist would find them a priori ridiculous. Take our case here, if the hypothesis tested in our example is something ridiculous such a power posing, would it matter whether we ran 1 or 1000 regressions? $\endgroup$ – statslearner Dec 16 '17 at 16:20
7
$\begingroup$

The statistical interpretation is much less clear than, what you are asking for, the mathematical treatment.

Mathematics is about clearly defined problems. E.g. rolling a perfect dice, or drawing balls from an urn.

Statistics is applied mathematics where the mathematics provides a guideline but is not the (exact) solution.

In this case it is obvious that circumstances play an important role. If we perform a regression and then calculate (mathematics) some p value to express the strength then what is the interpretation (statistics) and value of the p value?

  • In the case of the 1000 regressions performed by researcher 1 the result is much more weak since this type of situation occurs when we do not really have a clue and are just exploring the data. The p value is just an indication that there may be something.

    So the p value is obviously less worth in the regression performed by researcher 1. And if researcher 1 or somebody using the results of researcher 1 would like to do something with the regression then the p value needs to be corrected. (and if you thought the difference between researcher 1 and researcher 2 was not enough, just think about the multitude of ways that researcher 1 can to correct the p value for multiple comparisons)

  • In the case of the single regression performed by researcher 2 the result is much stronger evidence. But that is because the regression does not stand on it's own. We have to include the reasons why researcher 2 did only one single regression. This could be because he had good (additional) reasons to already believe that the single regression is a good model for the data.

  • The setting of the regressions performed by researcher 1 and 2 is much different, and it is not often that you encounter both at the same time for the same problem. If this is the case then either

    • researcher 2 was very lucky

      This is not so uncommon, and we should better correct for this when interpreting literature, as well we should improve the publishing of the total picture of research. If there are a thousand researchers like researcher 2, and we will only see one of them publish a success, then because we did not see the failures of the other 999 researchers we might mistakingly believe we did not have a case like researcher 1

    • researcher 1 was not so smart and did an incredibly superfluous search for some regression while he might have possibly known from the start that it should have been that single one, and he could have performed a stronger test.

      For outsiders who are smarter than researcher 1 (do not care about the additional 999 regressions from the start) and read about the work, they might give more strength to the significance of the results, however still not as strong as he would do for the outcome of researcher 2.

      While researcher 1 may have been too conservative when correcting for 999 superfluous additional regressions, we can not ignore the fact that the research was done in a vacuum of knowledge and it is much more likely to find a lucky researcher of the type 1 than the type 2.

An interesting related story: In astronomy, when they were planning a better instrument to measure the cosmic background with higher precision, there were researchers that argued to only release half the data. This because there is only one shot to gather data. Once all the regressions have been performed by the dozens of different researchers (and because of the incredible variation and creativity of the theorist, there is certainly some fit to every possible, random, bump in the data), there is no possibility to perform a new experiment to verify (that is, unless you are able to generate a whole new universe).

Improve this answer
$\endgroup$
19
  • 1
    $\begingroup$ +1 for @MartijnWeterings as I said in my comments the problem was not well posed mathematically. I got the impression that the OP thought that there is a paradox because because the two researchers would be led to the same model choice but the one doing 1000 regressions is penalized because of the need for a multiple comparison issue. I don't see this as a paradox at all (not clear but I think the OP did). You gave a very beautifully written and correct answer that also explains intuitively why the two cases are different. I think the OP should give your answer a check! $\endgroup$ – Michael R. Chernick Dec 14 '17 at 12:02
  • $\begingroup$ @MichaelChernick your comments could have been sufficient to handle the case/problem, but I found it usefull to state an answer with a strong 'statistics $\neq$ mathematics' flavour. Moving away from the necessary 'mathematical/theoretical justification', and acknowledging that statistical terms and problems are more vaguely defined than mathematical problems, makes the ambiguity in the question much more clear. $\endgroup$ – Sextus Empiricus Dec 14 '17 at 12:51
  • 1
    $\begingroup$ Also, I know this is common practice, but don't you find troubling to say one result is "stronger evidence" than the other, when they are the exact same model and data from the same data generating process? The only thing different is how much a third party looked at the data, and this should not have any relation with the DGP itself or your prior beliefs about the problem. Should researcher 2 analysis be tainted by researcher 1's ignorance, for instance? $\endgroup$ – statslearner Dec 14 '17 at 15:33
  • 1
    $\begingroup$ @MartijnWeterings why should the intention of the researcher matter for the interpretation of the data? If you're using this as a heuristic, as a layman interpreting an expert result, this is fine. But for a scientist analyzing the data, it seems the researcher's intention should have no bearing on your interpretation of the evidence. $\endgroup$ – statslearner Dec 14 '17 at 15:52
  • 1
    $\begingroup$ So it seems that you are using the researcher's behavior as a proxy for your prior. If the researcher ran 1000 regressions, that would correspond to a low prior to that specific hypothesis. If he ran just 1, this would correspond to a high prior on that hypothesis. If you had the two cases, then you don't know which prior to use. $\endgroup$ – statslearner Dec 15 '17 at 15:53
1
$\begingroup$

Short story: we don’t have enough information to answer your question because we don’t know anything about the methods used or the data collected.

Long answer...The real question here is whether each researcher is doing:

  • rigorous science
  • rigorous pseudoscience
  • exploration of data
  • data dredging or p-hacking

Their methods will determine the strength of the interpretation of their results. This is because some methods are less sound than others.

In rigorous science we develop a hypothesis, identify confounding variables, develop controls for variables outside our hypothesis, plan test methods, plan our analytical methodology, perform tests / collect data, and then analyze data. (Note that the analytical methods are planned before the test occurs). This is the most rigorous because we must accept data and analysis that does not agree with the hypothesis. It isn’t acceptable to change methods after the fact to get something interesting. Any new hypothesis from the findings have to go through the same process again.

In pseudoscience we often take data that is already collected. This is more difficult to use ethically because it is easier to add biases to the results. However, it is still possible to follow the scientific method for ethical analysts. It may be difficult to set up proper controls though and that needs to be researched and noted.

Exploration of data is not based on science. There is no specific hypothesis. There is not a priori evaluation of confounding factors. Also, it is difficult to go back and re-do the analysis using the same data, because the results may be tainted by prior knowledge or modeling and there is no new data to use for validation. A rigorous scientific experiment is recommended to clarify possible relationships found from exploratory analysis.

Data dredging or P-hacking is where an “analyst” performs multiple tests hoping for an unexpected or unknown answer or manipulates the data to get a result. The results may be simple coincidence, may be the result of confounding variable(s), or may not have have meaningful effect size or power.

There are some remedies for each problem, but those remedies have to be carefully evaluated.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I believe you are adding unnecessary noise to the question. Assume they used the best methods available. The data was not collected by them, but by a statistical agency, so they had no control over data collection. The only difference is how much each researcher explored the data. One of them explored a lot, the other explored only once. Both get the same final model with the same data. Should they make different inferences? And how should that affect your inference? $\endgroup$ – statslearner Dec 16 '17 at 4:44
  • $\begingroup$ This isn’t extra noise. Math is math. If the models are identical then they are identical. How you interpret the models is dependent on all of the other variables not included in your problem. If you ignore all of the other context and design or experiments the answer is simple, both models perform equally mathematically and are both scientifically weak. $\endgroup$ – Adam Sampson Dec 16 '17 at 5:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.