Does the order of magnitude of sample values influence p-values and effect size?

Question

When using an independent-samples t-test to compare two variables, do the absolute values of the samples affect the test statistic and the effect size?

In other words, is it easier to obtain significance if the two variables are e.g. in the range 0-10 (e.g. number of fingers raised by one person vs another) versus if they are in the order of e.g. thousands (e.g. number of neurons that fire per minute in one brain region vs another)? And similarly, assuming normal variability for the two variables, does effect size depend on the absolute values, or does only the relative difference between the two variables (which can be the same regardless whether the variables contain small or large numbers) matter?

Answer 1

What matters is how many standard deviations apart the means are. See what happens if you take two samples and either

i) add 1000 to both

ii) multiply both by 100

Notice that the t-statistic is unchanged in either case.

However, if you add 10 to one variable, that could change things substantively, if 10 wasn't a small fraction of the standard deviation of the difference in means.

Here's a t-test for two samples (equal variance assumption):

> t.test(x,y,var.equal=TRUE)

        Two Sample t-test

data:  x and y
t = -4.2271, df = 25, p-value = 0.000276

Note that $t = -4.2271$ and p-value $= 0.000276$

> t.test(x+1000,y+1000,var.equal=TRUE)

        Two Sample t-test

data:  x + 1000 and y + 1000
t = -4.2271, df = 25, p-value = 0.000276

Adding 1000 to each still gave $t = -4.2271$ and p-value $= 0.000276$

> t.test(x*100,y*100,var.equal=TRUE)

        Two Sample t-test

data:  x * 100 and y * 100
t = -4.2271, df = 25, p-value = 0.000276

Multiplying each by 100 still gave $t = -4.2271$ and p-value $= 0.000276$. But now see what adding 10 to one of them does:

> t.test(x,y+10,var.equal=TRUE)

        Two Sample t-test

data:  x and y + 10
t = -30.2065, df = 25, p-value < 2.2e-16

Completely different!

And similarly, assuming normal variability for the two variables, does effect size depend on the absolute values, or does only the relative difference between the two variables (which can be the same regardless whether the variables contain small or large numbers) matter?

That depends on how you defined "effect size". Some people would define it in the original units, others in terms of number of standard deviations or standard errors. In the physical sciences, the first makes more sense, in areas where the raw units don't mean much, the second makes more sense.

Answer 2

There seem to be multiple questions. I am trying to answer the first question and hope I am not confused about this myself.

In practice, different absolute values would affect the sample statistic of a t-test because they most likely affect the means and that is what you would generally be testing for. You may want to try the following R code to see the effect:

large_values <- rnorm(100,mean=1000, sd=4)
small_values <- rnorm(100, mean=0.4, sd=4)
middle_values <- rnorm(100, mean=10, sd=4)

t.test(large_values, small_values)
t.test(large_values, middle_values)

You could experiment a little by adjusting the mean to see when the p-value becomes larger then, say 0.05.

Note that it would theoretically be possible to conjure corner cases where the absolute values of a sample are choses in such a way that the test-statistic is unaffected. Consider the code below.

corner_100 = c(100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100)
corner_10 = c(10,10,10,10,10,10,10,10,10,10)
t.test(corner_100, corner_10)

I hope this both correct and helpful.

Answer 3

I think RndmSymbl is correct for the absolute values now lets talk about scale. As he said what matters is the $\mu$ and $\sigma$ that go into the t-test for each sample so we want to know how these quantities vary under a rescaling. For $\mu$ lets be rigorous:

Let $X$ be one of your samples and let $\bar{X}$ represent taking the mean. $f(\bar{X})=\bar{f(X)}$ if and only if $f$ is a linear function

A similar statement can be said for $\sigma^2$ if the function $f$ is purely scalar multiplication. This means that the values that go into your t-test are invariant under linear reparameterization. As long as you scale in this way your $p-value$ will be the same.

That said, I doubt your examples are purely linear rescalings since they are of a different nature entirely.