0
$\begingroup$

Found this question here, and I couldn't answser it and thought it would be a wonderful question for this site.

Here's a little thought experiment for your weekend pleasure. Consider the following:

Joe Scientist decides to conduct a study (call it Study A) to test the hypothesis that a parameter D > 0 vs. the null hypothesis that D = 0. He designs a study, collects some data, conducts an appropriate statistical analysis and concludes that D > 0. This result is published in the Journal of Awesome Results along with all the details of how the study was done.

Jane Scientist finds Joe's study very interesting and tries to replicate his findings. She conducts a study (call it Study B) that is similar to Study A but completely independent of it (and does not communicate with Joe). In her analysis she does not find strong evidence that D > 0 and concludes that she cannot rule out the possibility that D = 0. She publishes her findings in the Journal of Null Results along with all the details.

From these two studies, which of the following conclusions can we make?

  1. Study A is obviously a fraud. If the truth were that D > 0, then Jane should have concluded that D > 0 in her independent replication.

  2. Study B is obviously a fraud. If Study A were conducted properly, then Jane should have reached the same conclusion.

  3. Neither Study A nor Study B was a fraud, but the result for Study A was a Type I error, i.e. a false positive.

  4. Neither Study A nor Study B was a fraud, but the result for Study B was a Type II error, i.e a false negative.

I realize that there are a number of subtle details concerning why things might happen but I've purposely left them out. My question is, based on the information that you actually have about the two studies, what would you consider to be the most likely case? What further information would you like to know beyond what was given here?

$\endgroup$
  • 1
    $\begingroup$ I think the question is too vague to answer. For instance, how may we interpret "along with all the details of how the study was done."? Do the details include standard errors, or an assessment of model-fit in general? $\endgroup$ – Stijn May 21 '13 at 8:53
  • $\begingroup$ The vagueness seems to be intentional. I think a technical answer based on standard errors or model fit would not be particularly interesting. The point is to force us to lay out hidden assumptions about research practices, publication bias, power and effect sizes and how they interact together. Not sure if this is really relevant on this site, though. $\endgroup$ – Gala May 21 '13 at 10:28
3
$\begingroup$

Similar to Stijn's comment, there isn't sufficient information to make a conclusive judgment. There could be fraud (in either of the papers, not just the first), or there could have been a Type I / Type II error in one of the papers.

Out of interest if the two studies were identical and independent, so had identical power functions which factored, would it make sense to compare (ruling out fraud) the scenario for $D=0$:

$P(A=false positive, B=true negative)=\alpha(1 -\alpha)$

Against $D>0$

$P(A=true positive, B=false negative)=(1 -\beta)\beta$

And going with the more likely scenario?

$\endgroup$
  • $\begingroup$ "Statistics means never having to say you're certain" $\endgroup$ – Henry Feb 21 '15 at 12:26
2
$\begingroup$

The information provided does not indicate fraud by either party. Assuming 95% confidence as the acceptance point then 2 in 20 tests could be expected to produce a type 1 or 2 error. This is a major reason for doing meta analyses.

Furthermore, to check for fraud, one would want to review the record keeping for the two experiments and perhaps corroborative evidence from colleagues.

As well as natural variability, a major reason for obtaining different results is confounding, where for one reason or another the conditions or sample were not identical, or perhaps there were equipment or calibration problems.

$\endgroup$
  • 3
    $\begingroup$ I agree with this and other similar remarks. I don't think much purpose would be served by leaving this question in the forum under its present name. Anyone facing the difficult and delicate problem of detecting fraud, or of handling allegations of fraud, would not find it helpful or pertinent. $\endgroup$ – Nick Cox May 21 '13 at 12:25
  • $\begingroup$ What exactly is meta analysis? $\endgroup$ – MaoYiyi May 21 '13 at 14:34
  • $\begingroup$ @NickCox Please, change the title. I am not knowledgable enough to know how to properly label this. $\endgroup$ – MaoYiyi May 21 '13 at 14:35
  • $\begingroup$ My second thoughts are that the question should not be kept at all, so changing the title is secondary. On meta-analysis, do please Google. There are whole books on it. $\endgroup$ – Nick Cox May 21 '13 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.