5
$\begingroup$

I found this phrase on one of the web sites that publish reviewer's quotes (they suppose to be funny). One of the reviewers told to the authors of paper:

First, unless my statistics is failing me, a less than 1.0 SD is not significant.

I do not know the paper, but I know tags under this comment:

#you were right about one thing #yes your statistics is failing you #submission

What does it mean? As for me, reviewer told that the result is in less than 1 SD distance from the mean, that means that p-value is quite big and is not significant, so the reviewer was right, but the hash tag states the opposite.

Could somebody clarify what happens here, why the reviewer's comment is so short (as for me, there is no enough information) and what does it mean?

UPD: To whom who thinks that it is not clear what I am asking about. I am asking "Is there something that I miss in the information?" It happened with me once, when I heard "p-value of 0.05 means that you are wrong in approx 30% of the cases". It made no sense for me at first sight, but with the help of the community I understood an important concept. Unfortunately, it did not happen for this case.

$\endgroup$
7
  • $\begingroup$ I guess it depends on the sample size... $\endgroup$ Mar 14, 2016 at 13:13
  • $\begingroup$ But when your sample size increases, your SD (in most of the cases) decreases. Did they talk about, probably, effect size (difference in means)? It should be obvious, but I can not get it... =( $\endgroup$ Mar 14, 2016 at 13:16
  • 4
    $\begingroup$ Makes little sense to me owing to lack of context. Perhaps we're supposed to imagine the reviewer's confusing the standard deviation (of data) with the standard error (the standard deviation of a test statistic) as a measure of the distance of the observed value of a test statistic from its mean under the null hypothesis. $\endgroup$ Mar 14, 2016 at 13:56
  • 2
    $\begingroup$ I mean perhaps the reviewer's supposed to be confusing, in the context of testing that the data $x$ have a normal mean $\mu=\mu_0$, or something similar, $\frac{\bar{x} - \mu_0}{\sigma} < 1$ with $\frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}} <1$, where $\sigma$ is the std deviation of $x$ & $n$ the no. observations. That's a guess; it's not clear for me either. If the reviewer were in fact talking about the std deviation of a test statistic they might be perfectly correct. I suspect the post rings a bell for a particular research community - not necessarily for users of statistics in general. $\endgroup$ Mar 14, 2016 at 15:00
  • 1
    $\begingroup$ This does require us to infer some context, but, especially in light of @MattKrause's upvoted & accepted answer, this seems clear enough to me. $\endgroup$ Mar 2, 2017 at 19:20

3 Answers 3

6
$\begingroup$

The reviewer is apparently trying to use the standard deviation as some sort of ad-hoc statistical test, but this doesn't work. The reviewer's snarky "unless my statistics is failing me" comment is therefore funny (or maddening, if this gets your paper rejected).

Specifically, standard deviation tells us how spread out the values are around the mean value. However, in most hypothesis-testing situations, we are interested in determining the means of each group and whether these means differ. To do this, we need to determine how accurately we know the mean of each group, and the relevant statistic here is the standard error of the mean, not the standard deviation. (These are related, in that $\textrm{se}_\textrm{mean} = \frac{s}{\sqrt{n}}$, where $s$ is the sample standard deviation and $n$ is the number of samples). In a $t$-test, the denominator is often the standard error of the mean (or something like it for two-sample t-tests).

It's easy simulate data that shows it is possible to find significant differences between groups that have a standard deviation of at least one. For example,

a = rnorm(100, 0, 1)  # Draws 100 random points from N(0,1)
b = rnorm(100, 1, 1)  # Draws 100 random points from N(1,1)
t.test(a, b)
#   Welch Two Sample t-test
#
#   data:  a and b
#   t = -8.0116, df = 190.746, p-value = 1.097e-13
#   alternative hypothesis: true difference in means is not equal to 0
#   95 percent confidence interval:
#     -1.4188010 -0.8581969
#   sample estimates:
#     mean of x  mean of y 
#     -0.1154137  1.0230852 

Similarly, you can imagine a scenario in which the data where each standard error of the mean is much larger than one, yet a $t$-test finds a significant difference. For example, suppose the means were -100 and 100: with enough data, we'd be able to tell them apart even if the standard deviation were large (e.g., $\sigma=30$ for $n=100$)

In summary, the reviewer has totally failed at statistics, which makes his/her snarky comment funny.

$\endgroup$
3
  • $\begingroup$ Yes, but do you mean that we can re-write: "First, unless my statistics is failing me, a SD less than 1.0 is not significant"? I thought that the comment means "First, unless my statistics is failing me, $ < 1.0 \cdot \sigma$ is not significant"...$se$ will always be less than 1.0 SD for any reasonable sample size, so it is strange that reviewer made this mistake. $\endgroup$ Mar 14, 2016 at 14:24
  • 3
    $\begingroup$ The SD has no direct connection to a significance test, except through the SE, and there's no magical significance threshold at SE=1. More broadly, the whole point of that site is that reviewers sometimes say obnoxious, baffling, or downright stupid things and the post you quoted is an example of all three. I wouldn't look there for advice about academic writing or stats; it's meant to let people blow off steam. $\endgroup$ Mar 14, 2016 at 14:31
  • $\begingroup$ Thank you! Yes, now it is clear. I thought there is more sophisticated statistical issue in this post, but overestimated the smartness of the reviewer. $\endgroup$ Mar 14, 2016 at 14:33
2
$\begingroup$

It's unclear without more information. Here's an example where the standard deviation is less than 1.0, but the test is significant:

# draw 100 samples from two random Normal distributions with small standard deviations,
#  with different means:
x  <- rnorm(100, 1, .5)
x2 <- rnorm(100, 3, .5)

# note the standard deviation of the difference in means that will be tested:
sd(x-x2)

# here's the significant result:
t.test(x, x2)

Does that help?

$\endgroup$
4
  • $\begingroup$ Yes, it may be an explanation. However, I read "less than 1.0 SD" as $< 1.0 \cdot \sigma$, not $\sigma < 1.0$. $\endgroup$ Mar 14, 2016 at 13:38
  • $\begingroup$ My mistake! I misread that. $\endgroup$ Mar 14, 2016 at 13:40
  • $\begingroup$ No, you may be right and it may be an answer. But it would be completely stupid for the reviewer. $\endgroup$ Mar 14, 2016 at 13:41
  • $\begingroup$ I guess it's difficult to guess what the reviewer was meaning (they probably didn't know themselves). As such it's hard to come up with a definite answer...fun website though. $\endgroup$ Mar 14, 2016 at 14:10
1
$\begingroup$

Most likely the reviewer means that your parameter is closer than one standard deviation to the null hypothesis value. For instance, your regression slope coefficient is 0.5 while its standard deviation is 1. I emphasized to assume that the reviewer is using the correct standard deviation, i.e. adjusted to the sample size blah-blah. In this case if you're testing whether there is a slope in the regression, you compare 0.5 to 0, and observe that it's closer than the standard deviation.

For instance, consider gamma distribution with parameters $\alpha=0.1$ and $\beta=10000$, its mean is 1000 and $\sigma=3162$. If you test the distance between the mean and 1e-10 then the one tailed test would give you 5% significance. It's very skewed distribution.

Can you construct an example where p-value is significant while the distance is less than the standard deviation? Yes, of course. However, in most regressions this simple heuristic based on the standard deviations works fine. Moreover, to determine significance, you usually have to assume the probability distribution. While a simple comparison to the standard deviation does not require any assumptions on distributions. Therefore, I'd side with your reviewer in this case. Unless you have reasons to believe that she can't calculate the appropriate standard deviation of the parameter, which is unlikely (why would a reviewer be incompetent?).

Also, in applied work folks (like myself) speak in terms of standard deviations all the time. For instance, when talking about the accuracy of an instrument or economic significance of the coefficient. It's a very good measure of the dispersion.

$\endgroup$
4
  • $\begingroup$ How can $p$-value of a regression coefficient $\beta$ be significant (I guess you mean $p<0.05$ that $\beta=0$) if standard error of $\hat \beta$ is larger than $\hat \beta$? $\endgroup$
    – amoeba
    Mar 2, 2017 at 21:52
  • $\begingroup$ @amoeba, maybe a very skewed distribution and one tailed test? I need to think about this $\endgroup$
    – Aksakal
    Mar 3, 2017 at 0:11
  • $\begingroup$ Isn't your first paragraph describing the standard error of the parameter (i.e., its SD "corrected" by the square root of N) and not its raw standard deviation? With enough data, I'd expect it would be possible to confidently test a coefficient of 0.5 with an SD of 1.0 against a null hypothesis of zero and reject it. The higher spread may make the interpretation more complicated, but it certainly doesn't rule out significance. $\endgroup$ Mar 3, 2017 at 0:45
  • $\begingroup$ @MattKrause, you don't always have enough data. In physics I often had enough data, in business it depends. In marketing you often can get a lot of data, in economics almost never. $\endgroup$
    – Aksakal
    Mar 3, 2017 at 0:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.