2
$\begingroup$

I have two samples, say sample A and B, and a negative control.

I expect that the sample A to be different than the negative control, so I use a t-test to check for that.

Then, I expect sample B not to be different from the negative control. How do I test for that? I could use a t-test for that too and treat a high p value as confirmation, but I'm wondering if there is a better way?

$\endgroup$
  • $\begingroup$ How equal do you want them to be? The range of values for different was infinite. Are you proposing to assess the probability of exact equality? $\endgroup$ – John Jun 26 '13 at 2:28
6
$\begingroup$

One way to approach this question is called “equivalence testing”. You can look up information on that but generally speaking you cannot prove that two means are exactly equal (nor should you expect them to be in most cases) so you first have to define an equivalence margin (i.e. a difference that you consider negligible in practice).

Importantly, as Maarten already mentioned, a t-test is used to compare population means. It does not tell you if the two distributions are the same, if the difference is big, if there is a lot of overlap or anything like that buy only if the observed difference in means is likely to have come about purely through sampling variability. It also does not test if samples are different (to determine that, you just need to look at your data directly). The whole point of statistical inference is that your sample is a noisy representation of a broader population. Generally speaking, you should not expect statistics to tell you if something is “true” or “real” but only to help you deal with random variation.

$\endgroup$
3
$\begingroup$

To answer your question: not really. Statistics is very good at rejecting or not rejecting hypotheses, but confirming hypotheses is very hard (some say impossible). This is more a philosiphical than a statistical problem. An accessible introduction to this issue can be found in:

Ian Hacking (2001) "An Introduction to Probability and Inductive Logic". Cambridge: Cambridge University Press.


As an aside: With a t-test you are not testing whether the samples are equal, but only whether or not the means are equal. Below is a silly example using Stata: the three variables x, y, and z are clearly from different distributions, but we (correctly) cannot reject the hypothesis that the means are equal.

. clear

. set obs 10000
obs was 0, now 10000

. gen x = rnormal()

. 
. gen u = runiform()

. gen y = (u <  .5)*( rnormal() - 2 ) + ///
>         (u >= .5)*( rnormal() + 2 )

.                 
. gen z = rchi2(2) - 2

.                 
. twoway kdensity x || ///
>        kdensity y || ///
>        kdensity z

enter image description here

. 
. ttest x = y, unpaired unequal

Two-sample t test with unequal variances
------------------------------------------------------------------------------
Variable |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       x |   10000   -.0169198     .010027    1.002698   -.0365747    .0027351
       y |   10000   -.0180979     .022326    2.232599   -.0618613    .0256655
---------+--------------------------------------------------------------------
combined |   20000   -.0175089    .0122368    1.730549   -.0414941    .0064763
---------+--------------------------------------------------------------------
    diff |            .0011781    .0244743               -.0467948     .049151
------------------------------------------------------------------------------
    diff = mean(x) - mean(y)                                      t =   0.0481
Ho: diff = 0                     Satterthwaite's degrees of freedom =    13875

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.5192         Pr(|T| > |t|) = 0.9616          Pr(T > t) = 0.4808

. ttest x = z, unpaired unequal

Two-sample t test with unequal variances
------------------------------------------------------------------------------
Variable |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
       x |   10000   -.0169198     .010027    1.002698   -.0365747    .0027351
       z |   10000    .0086713    .0198435     1.98435   -.0302259    .0475686
---------+--------------------------------------------------------------------
combined |   20000   -.0041242    .0111166    1.572121   -.0259136    .0176652
---------+--------------------------------------------------------------------
    diff |           -.0255912     .022233               -.0691705    .0179882
------------------------------------------------------------------------------
    diff = mean(x) - mean(z)                                      t =  -1.1510
Ho: diff = 0                     Satterthwaite's degrees of freedom =  14792.6

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.1249         Pr(|T| > |t|) = 0.2497          Pr(T > t) = 0.8751
$\endgroup$
1
$\begingroup$

You design your study to have sufficient "statistical power" to detect a meaningful difference. This requires that you pre-specify the size of a difference between the "negative control" and treatment B that you would consider "small" or unimportant. You then back-calculate to see how probable it would be to find a null result (fail to reject the null) if the true value is at least that large. You want the probability to be small and you choose your sample size to make it sufficiently small, If the design goal of testing similarity of treatment B to the control is an importance design criterion, it would be wise to make the probability of such a failure 10% or lower. You would then say you have 90% "power" to detect an important difference. There are many places you can find formulas to calculate this sample size, although for complex designs where you may be controlling for confounding factors, it may require simulation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.