3
$\begingroup$

A question on how to prove that differences in mean value are statictically significant, and not just random noise.

I have a set of two observations, one of which I'm going to deliberately bias like this (in R):

m <- cbind( A=sample(0:10, 10, replace=TRUE), B=sample(1:11, 10, replace=TRUE) )

Now, even with such a small data set, the mean of B is usually (but not always) higher than that of A.

colMeans(m)

As I accumulate more measurements, the bias should be more clearly revealed.

Assuming I didn't know about the bias beforehand, how can I prove it mathematically (in R if possible)?

$\endgroup$
1
  • 1
    $\begingroup$ How about a two-sample t-test? $\endgroup$
    – Michael M
    May 30, 2014 at 9:05

2 Answers 2

5
$\begingroup$

As Michael Mayer said, a two-sample t-test is one approach. The test compares the means of two groups under the assumption that both samples are random, independent, and come from normally distributed population with unknown but equal variances. You have to think about whether this assumption makes sense for your situation.

In R:

> set.seed(1)
> m <- cbind( A=sample(0:10, 10, replace=TRUE), B=sample(1:11, 10, replace=TRUE) )
> colMeans(m)
  A   B 
5.5 6.6 
> t.test(m[,1],m[,2], var.equal=TRUE, paired=FALSE)

    Two Sample t-test

data:  m[, 1] and m[, 2]
t = -0.7802, df = 18, p-value = 0.4454
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -4.06206  1.86206
sample estimates:
mean of x mean of y 
      5.5       6.6 

Notice the p-value is large here (0.4454 because we don't have a large sample (so less information). However:

> m <- cbind( A=sample(0:10, 1000, replace=TRUE), B=sample(1:11, 1000, replace=TRUE) )
> colMeans(m)
    A     B 
4.984 5.866 
> t.test(m[,1],m[,2], var.equal=TRUE, paired=FALSE)

    Two Sample t-test

data:  m[, 1] and m[, 2]
t = -6.1175, df = 1998, p-value = 1.14e-09
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.1647501 -0.5992499
sample estimates:
mean of x mean of y 
    4.984     5.866 

and we have a very small p-value: 1.14e-09. The p-value is smaller due to a larger sample size (i.e. more information) and hence more confidence to reject $H_0$

$\endgroup$
0
$\begingroup$

Many thanks for the quick and thorough answers.

It turns out that what I needed is a one-sample t test.

I had worked out a model in which I expected the results to average out at a certain mean value.

With few measurements, it's uncertain whether the actual mean differs from the expected mean by chance or because of a real effect. This usually produces a high p-value.

With many measurements, the one-sample t test can produce low p-values, indicating that it's statistically likely that there is a real difference.

E.g. with a low number of samples, the following typically produces high p-values, indicating that the hypothetical mean of 102 could have come about from the sample:

t.test(rnorm(10, mean=100, sd=5), mu=102)

As the number of samples increases, lower p-values start to dominate, indicating that it's likely the sample really does have a mean different from the hypothetical value.

t.test(rnorm(100, mean=100, sd=5), mu=102)

Big samples consistently yield 2.2e-16, meaning that it's virtually certain the real mean is not 102.

t.test(rnorm(1000, mean=100, sd=5), mu=102)
$\endgroup$
1
  • $\begingroup$ Please, visit our Help Center to merge your accounts and register once and for all. $\endgroup$
    – chl
    May 31, 2014 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.