9
$\begingroup$

I've encountered this sentence while reading an article on sciencemag.org.

In the end, responses from just 7600 researchers in 12 countries were included because the remaining data were not considered statistically significant.

Is this a proper way to do research? To leave out results because they were not considered statistically significant?

$\endgroup$
  • $\begingroup$ A reader notes that a report is available at eurodoc.net/index.php?s=file_download&id=122 and refers us specifically to pp. 104-5 of the full report (but I don't know where that is available). $\endgroup$ – whuber Sep 30 '11 at 20:44
  • 1
    $\begingroup$ That's the link for the full report. $\endgroup$ – Aaron - Reinstate Monica Sep 30 '11 at 20:52
  • $\begingroup$ @Aaron Thanks. For some reason, my first attempt produced only a 6-page summary, but now the link works fine. $\endgroup$ – whuber Oct 3 '11 at 18:51
6
$\begingroup$

In the report cited in whuber's comment, it says on page 104 [pg 114 in the pdf]:

The survey succeeded in activating the participation of approximately 8,900 doctoral candidates from more than 30 countries...

Then, spanning pages 104-105, it says:

While conducting data cleaning procedures, the Eurodoc survey experts' team decided to run a power test analysis. Based on the assumption of fully completed questionnaires which will result in a multi normal distribution, a power test for estimation of the confidence interval was used. This was done to test the accuracy of the data. It was decided to accept maximum a 6% error-level at a 95% confidence interval. A loss of 16% of the sampling size resulted in a sample of 12 participating countries with 7,600 participants.

So it's not really clear exactly why the 16% loss in the sample, but the assumption of incomplete responses is likely correct. (And you can see why the reporter was confused.)

$\endgroup$
  • 5
    $\begingroup$ The paragraph from pages 104-105 speaks for itself in documenting nonsense. $\endgroup$ – Frank Harrell Oct 2 '11 at 10:50
13
$\begingroup$

That sentence does not actually make sense and is clearly in error.

Data cannot be statistically significant or insignificant. Only relationships between data, the product of statistical tests, can be spoken about in these terms.

If the question is: Can we drop data from our analyses because the inclusion of that data means we cannot reject the null hypothesis? The answer is — obviously, I hope! — no. The message you've cited is a news report, not a scientific paper. Had it been a paper that was reviewed, it never would have gotten in.

Probably, data was not included because there are substantive reasons to not include those data. Probably, as others have suggested, the excluded data was incomplete or collected using different or incomparable methods.

$\endgroup$
  • $\begingroup$ Hope so... ;) . $\endgroup$ – FairMiles Feb 13 '13 at 19:25
7
$\begingroup$

No.

I suspect the reporter meant to say that the other individuals were left out because the surveys were incomplete or internally inconsistent.

$\endgroup$
  • 1
    $\begingroup$ that's also what I suspected, but this sentence is clearly misleading.. $\endgroup$ – upabove Sep 30 '11 at 9:58
2
$\begingroup$

No, but reporters can use technical jargon completely nonsensically.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.