10
$\begingroup$

I've encountered this sentence while reading an article on sciencemag.org.

In the end, responses from just 7600 researchers in 12 countries were included because the remaining data were not considered statistically significant.

Is this a proper way to do research? To leave out results because they were not considered statistically significant?

Improve this question
$\endgroup$
3
  • $\begingroup$ A reader notes that a report is available at eurodoc.net/index.php?s=file_download&id=122 and refers us specifically to pp. 104-5 of the full report (but I don't know where that is available). $\endgroup$
    – whuber ♦
    Commented Sep 30, 2011 at 20:44
  • 1
    $\begingroup$ That's the link for the full report. $\endgroup$
    – Aaron - mostly inactive
    Commented Sep 30, 2011 at 20:52
  • $\begingroup$ @Aaron Thanks. For some reason, my first attempt produced only a 6-page summary, but now the link works fine. $\endgroup$
    – whuber ♦
    Commented Oct 3, 2011 at 18:51

4 Answers 4

Reset to default
6
$\begingroup$

In the report cited in whuber's comment, it says on page 104 [pg 114 in the pdf]:

The survey succeeded in activating the participation of approximately 8,900 doctoral candidates from more than 30 countries...

Then, spanning pages 104-105, it says:

While conducting data cleaning procedures, the Eurodoc survey experts' team decided to run a power test analysis. Based on the assumption of fully completed questionnaires which will result in a multi normal distribution, a power test for estimation of the confidence interval was used. This was done to test the accuracy of the data. It was decided to accept maximum a 6% error-level at a 95% confidence interval. A loss of 16% of the sampling size resulted in a sample of 12 participating countries with 7,600 participants.

So it's not really clear exactly why the 16% loss in the sample, but the assumption of incomplete responses is likely correct. (And you can see why the reporter was confused.)

Improve this answer
$\endgroup$
1
  • 5
    $\begingroup$ The paragraph from pages 104-105 speaks for itself in documenting nonsense. $\endgroup$
    – Frank Harrell
    Commented Oct 2, 2011 at 10:50
13
$\begingroup$

That sentence does not actually make sense and is clearly in error.

Data cannot be statistically significant or insignificant. Only relationships between data, the product of statistical tests, can be spoken about in these terms.

If the question is: Can we drop data from our analyses because the inclusion of that data means we cannot reject the null hypothesis? The answer is — obviously, I hope! — no. The message you've cited is a news report, not a scientific paper. Had it been a paper that was reviewed, it never would have gotten in.

Probably, data was not included because there are substantive reasons to not include those data. Probably, as others have suggested, the excluded data was incomplete or collected using different or incomparable methods.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hope so... ;) . $\endgroup$
    – FairMiles
    Commented Feb 13, 2013 at 19:25
7
$\begingroup$

No.

I suspect the reporter meant to say that the other individuals were left out because the surveys were incomplete or internally inconsistent.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ that's also what I suspected, but this sentence is clearly misleading.. $\endgroup$
    – upabove
    Commented Sep 30, 2011 at 9:58
2
$\begingroup$

No, but reporters can use technical jargon completely nonsensically.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.