1
$\begingroup$

How do I work out the p value if the variance of my control and experimental group are different? I can't do the t-test anymore. How do I work the p -value out, what method? Also, my two groups are different amount of trials. The control has 25 values and the experimental only has 5 values. Hence I can't do the chi-squared test.

$\endgroup$
1
  • 2
    $\begingroup$ Welcome to CV. You are referring to two tests with very different hypotheses. What are you trying to demonstrate? $\endgroup$ Sep 15, 2018 at 12:16

2 Answers 2

3
$\begingroup$

As you are talking about the t-test, I assume that you want to compare the mean of the experimental and the control group. An 'extension' of Student's t-test to unequal sample sizes and unequal variances is Welch's t-test. Welch's t-test uses the test statistic

$$ \frac{\overline{X}_1 - \overline{X}_2}{\sqrt{ \; {s_1^2 \over N_1} \; + \; {s_2^2 \over N_2}}}. $$

This test statistic is approximately t-distributed with $\nu$ degrees of freedom, with $\nu$ given by:

$$ {{\left( \; {s_1^2 \over N_1} \; + \; {s_2^2 \over N_2} \; \right)^2 } \over { \quad {s_1^4 \over N_1^2 (N_1-1)} \; + \; {s_2^4 \over N_2^2 (N_2-1) } \quad }} .$$

The p-value is then calculated using Student's t-distribution with $\nu$ degrees of freedom.

$\endgroup$
4
  • $\begingroup$ The degrees of freedom you present is Satterthwaite's formula. Welch's formula is $$df = -2 + \frac{\left(\frac{s_{x}^{2}}{n_{x}} + \frac{s_{y}^{2}}{n_{y}}\right)^{2}}{\frac{\left(s_{x}^{2}\right)^{2}}{n_{x}+1} + \frac{\left(s_{y}^{2}\right)^{2}}{n_{y}+1}}$$ Satterthwaite, F. E. 1946. An approximate distribution of estimates of variance components. Biometrics Bulletin 2: 110-114. Welch, B. L. 1947. The generalization of “Student’s” problem when several different population variances are involved. Biometrika 34: 28-35. $\endgroup$
    – Alexis
    Sep 15, 2018 at 18:58
  • $\begingroup$ @Alexis Thank you. Wikipedia calls it the Satterthwaite-Welch formula. $\endgroup$
    – Nussig
    Sep 15, 2018 at 19:19
  • 1
    $\begingroup$ Yes well, there's Wikipedia, then there's the actual authors' respective work cited. :) $\endgroup$
    – Alexis
    Sep 15, 2018 at 19:39
  • $\begingroup$ Formulas from original papers sometimes get adjusted or refined in usage: Snedecor and Cochran (1980) and the NIST handbook (current) both have what looks to me like the same formula as Wikipedia. Would be the last to claim Wikipedia is always right, but maybe this time.... $\endgroup$
    – BruceET
    Sep 15, 2018 at 21:54
1
$\begingroup$

When sample sizes are very different, as in your case, a problematic situation occurs for the pooled two-sample t test if the population variance is larger in the population from which we have the smaller sample size.

Let's look at an example in which we have 25 observations x from $\mathsf{Norm}(50, 2)$ and 5 observations y from $\mathsf{Norm}(60, 5),$ where the second argument in my notation for a normal distribution is the population standard deviation.

Here are fake data, generated in R, and their stripcharts:

set.seed(915);  x = rnorm(25, 50, 2);  y = rnorm(5, 60, 5)
all = c(x,y);  gp = c(rep(1,25), rep(2,5))
stripchart(all ~ gp, ylim=c(.5, 2.6), pch="|")

enter image description here

In R, the default version of a two-sample t test is the Welch test:

t.test(all ~ gp)

    Welch Two Sample t-test

data:  all by gp
t = -2.7021, df = 4.1464, p-value = 0.05195
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -14.43014179   0.09579794
sample estimates:
mean in group 1 mean in group 2 
       50.11499        57.28217 

So at the 5% level, the difference in sample means (50.11 and 57.28) is not quite significantly different from $0.$ [The P-value for one-sided test against alternative $H_a: \mu_t < \mu_c$ would have been half as large, significant at the 5% level.]

The P-value for this two-sided test is found in R (where pt is the CDF of a t distribution) as follows:

2*pt(-2.7021, 4.1464)
[1] 0.05194632

Ordinarily, you need software to find an exact P-value. Taking df = 4, you can bracket the P-value corresponding to $|T| = 2.7$ by using a printed table of t distributions: Noting that the probability in the right-hand tail of $\mathsf{T}(\nu=4)$ beyond 2.776 is 0.025, and that the probability beyond 2.131 is 0.05, you can deduce that the P-value for the two-sided test is between 0.05 and 0.10.

With $n_1 = 25,\, n_2 = 5,$ the degrees of freedom $\nu$ from the formula given in @Nussig's Answer (+1) must lie between $\min(n_1-1, n_2 - 1) = 4$ and $n_1 + n_2 - 2 = 28.$ In our example, we happen to be near the lowest possible value of $\nu.$ [Some software programs round to integer degrees of freedom.]

Notes: (1) An inadvisable pooled two-sample t test (with t.test(all ~ gp, var.eq=T)) uses $\nu = n_1 + n_2 - 2$ along with a somewhat different formula for the t statistic $(T = -5.298),$ giving the (bogus) 'highly significant' P-value 1.228e-05.

(2) When $n_1 = 25, n_2 = 5, \sigma_1 = 2, \sigma_2 = 5,$ a pooled two-sample t test at 'advertised' significance level 5% has actual significance level above 25%. The following simulation verifies this:

set.seed(918) 
pv = replicate(10^5, t.test(rnorm(25,0,2),rnorm(5,0,5),var.eq=T)$p.value)
mean(pv < .05)
[1] 0.26253

By contrast, the Welch two-sample t test has nearly the nominal 5% level:

set.seed(2018)
pv = replicate(10^5, t.test(rnorm(25,0,2),rnorm(5,0,5))$p.value)
mean(pv < .05)
[1] 0.05404
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.