10
$\begingroup$

Imagine I work in a screw factory (which is not true). We have been producing a series of 5cm-long (on average) screws for years. I want to modify the production parameters in order to save some money and then check if my new screws are shorter, longer or of equal lengths (5cm).

So I make a series with my new parameters and then I collect $n$ screws this new production and measure their lengths. I calculate the mean $\bar x$ and the standard deviation $s$. With these info, I calculate the the standard error $SE = \frac{s}{\sqrt{n}}$ and calculate the p-value, $p$.

How can I calculate the effect size in this experiment? What would be the intuitive meaning of the effect size here?

Improve this question
$\endgroup$
1
  • $\begingroup$ If you are truly interested in the "of equal lengths," bit (within some level of tolerance) you need to conduct a test for equivalence as well as the two-sided test for difference you and @gung are connecting over. The two one-sided tests framework for conducting equivalence tests builds straightforwardly off of the basic t test. TOST software in Stata here. $\endgroup$
    – Alexis
    Sep 23, 2014 at 23:56

2 Answers 2

Reset to default
9
$\begingroup$

The standard effect size for a one-sample t-test is the difference between the sample mean and the null value in units of the sample standard deviation:
$$ d = \frac{\bar x - \mu_0}{s} $$ The interpretation here is essentially the same as for the two-sample version of the standardized mean difference: it is the number of standard deviations that your distribution diverges on average. As in most cases with effect sizes, you can think of it as taking the $N$ out of your test statistic. Thus, with a test statistic / $p$-value you get a sense of the confidence you have in your result, but these conflate the size with $N$, so from a small $p$ you don't know if you have a big effect with a small $N$ or a small effect with a big $N$. Here, you would get a point estimate of the magnitude of the shift, but you don't know from $d = .5$ whether or not you can be confident that the true effect isn't $0$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ (+1) Does this quantity have a name? In the two-sample case, it's Cohen's d. How should one call it in one-sample case? I am thinking of a brief reporting in parentheses in an article text, along the lines of "value X was larger in condition A than in B (p=0.001, two-sample t-test, n=20, Cohen's d=0.5)". In one-sample case I'd like to write something like "value X was larger than 0 (p=0.001, one-sample t-test, n=20, ??? d=0.5)", but I am not sure how to call d. $\endgroup$
    – amoeba
    Feb 5, 2019 at 22:20
  • 1
    $\begingroup$ @amoeba, you could call it $d$, if you wanted, but it isn't Cohen's $d$. More generally, these are 'standardized mean differences'. In the 2 sample case it's the standardized difference between the two means; here it's the standardized difference between the mean and the reference value. I don't know of an official name, but calling it 'd' by analogy seems perfectly reasonable. I just don't think that Cohen can lay claim to it, per se. $\endgroup$
    – gung - Reinstate Monica
    Feb 6, 2019 at 1:21
3
$\begingroup$

The important consideration is "what measure of effect matters for your purposes?".

One common approach is to measure number of standard deviations of shift, that's probably not much use to you if you're trying to work out how much the mean changed. After all, if you're manufacturing screws, knowing the screws are 0.073 standard deviations shorter is not much use. Knowing they're about 0.016mm shorter on average -- that potentially matters.

But if you were dealing with things whose scales are somewhat arbitrary, the numbers themselves aren't especially inherently meaningful, and then sd's of shift makes far more sense. If you're dealing with something like test scores, a change of 3 doesn't necessarily mean anything very much... but if you know it's half a standard deviation, that might matter much more.

Improve this answer
$\endgroup$
3
  • $\begingroup$ These are good points. I took the example as just a motivating story, where the real point was to get at the effect size for a one-sample t-test, & I assumed (but perhaps incorrectly) that a standardized effect size was being asked about. But in this case, .016mm would definitely be more useful. $\endgroup$
    – gung - Reinstate Monica
    Sep 24, 2014 at 3:00
  • $\begingroup$ @gung I don't mean to suggest there's anything wrong with your answer - it's a good answer to the question. It's no doubt the most common way to measure effect size for the one sided test, and probably also what the OP actually wants. But perhaps not what at least some other readers will need. I thought I should explain that what you do depends on your needs/what makes sense in the context of the individual problem, not on what the most common thing other people do is. The problem determines the statistics we should use, not the other way around. $\endgroup$
    – Glen_b
    Sep 24, 2014 at 3:05
  • 1
    $\begingroup$ Of course, I didn't mean to come off as suggesting you had implied criticism. I'm not the least offended. These are good points & I'm glad they have been added to the thread. $\endgroup$
    – gung - Reinstate Monica
    Sep 24, 2014 at 3:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.