10
$\begingroup$

I have two samples of data, a baseline sample, and a treatment sample.

The hypothesis is that the treatment sample has a higher mean than the baseline sample.

Both samples are exponential in shape. Since the data is rather large, I only have the mean and the number of elements for each sample at the time I will be running the test.

How can I test that hypothesis? I'm guessing that it is super easy, and I've come across several references to using the F-Test, but I'm not sure how the parameters map.

$\endgroup$
  • 2
    $\begingroup$ Why don't you have the data? If the samples are really large non-parametric tests should work great, but it sounds like you're trying to run a test from the summary statistics. Is that right? $\endgroup$ – Mimshot Nov 15 '13 at 20:56
  • $\begingroup$ Are the baseline and treatment values from the same patient set or are the two groups independent? $\endgroup$ – Michael M Nov 15 '13 at 21:11
  • 1
    $\begingroup$ @Mimshot, the data is streaming, but you are correct that I'm trying to run a test from the summary statistics. It works quite well with a Z test for normal data $\endgroup$ – Jonathan Dobbie Nov 15 '13 at 21:14
  • 1
    $\begingroup$ Under these circumstances, an approximate z-test is maybe the best you can do. However, I would more care about how large the true treatment effect is, not about statistical significance. Remember, that with large enough samples, any tiny true effect will lead to a small p value. $\endgroup$ – Michael M Nov 15 '13 at 21:22
  • 1
    $\begingroup$ @january - although, if his sample sizes are large enough, by the CLT they will be very close to normally distributed. Under the null hypothesis, the variances would be the same (as the means are), so, with a large enough sample size, a t-test should work fine; it won't be as good as you can do with all the data, but still would be OK. $n_1 = n_2 = 100$, for example, would be pretty good. $\endgroup$ – jbowman Nov 15 '13 at 21:37
14
$\begingroup$

You can test equality of the mean parameters against the alternative that the mean parameters are unequal with a likelihood ratio test (LR test). (However, if the mean parameters do differ and the distribution is exponential, this is a scale shift, not a location shift.)

For a one-tailed test (but only asymptotically in the two tailed case), I believe that the LR test comes out to be equivalent to the following (to show that this is in fact the same as the LR test for the one-tailed case one would need to show the LR statistic was monotonic in $\bar x/\bar y$):

Let's say we parameterize the $i$th observation in the first exponential as having pdf $1/\mu_x \exp(-x_i/\mu_x)$ and the $j$th observation in the second sample as having pdf $1/\mu_y \exp(-y_j/\mu_y)$ (over the obvious domains for the observations and parameters).
(To be clear, we're working in the mean-form not the rate-form here; this won't affect the outcome of the calculations.)

Since the distribution of $X_i$ is a special case of the gamma, $\Gamma(1,\mu_x)$, the distribution of the sum of $X$'s, $S_x$ is distributed $\Gamma(n_x,\mu_x)$; similarly that for the sum of the $Y$s, $S_y$ is $\Gamma(n_y,\mu_y)$.

Because of the relationship between gamma distributions and chi-squared distributions, it turns out that $2/\mu_x S_x$ is distributed $\chi^2_{2n_x}$. The ratio of two chi-squares on their degrees of freedom is F. Hence the ratio, $\frac{\mu_y}{\mu_x}\frac{S_x/n_x}{S_y/n_y} \sim F_{2n_x,2n_y}$.

Under the null hypothesis of equality of means, then, $\bar x/\bar y \sim F_{2n_x,2n_y}$, and under the two sided alternative, the values might tend to be either smaller or larger than a value from the null distribution, so you need a two-tailed test.


Simulation to check that we didn't make some simple mistake in the algebra:

Here I simulated 1000 samples of size 30 for $X$ and 20 for $Y$ from an exponential distribution with the same mean, and computed the above ratio-of-means statistic.

Below is a histogram of the resulting distribution as well as a curve showing the $F$ distribution we computed under the null:

simulated example distribution of ratio statistic under the null


Example, with discussion of computation of two-tailed p-values:

To illustrate the calculation, here's two small samples from exponential distributions. The X-sample has 14 observations from a population with mean 10, the Y-sample has 17 observations from a population with mean 15:

x: 12.173  3.148 33.873  0.160  3.054 11.579 13.491  7.048 48.836 
   16.478  3.323  3.520  7.113  5.358

y:  7.635  1.508 29.987 13.636  8.709 13.132 12.141  5.280 23.447 
   18.687 13.055 47.747  0.334  7.745 26.287 34.390  9.596

The sample means are 12.082 and 16.077 respectively. The ratio of means is 0.7515

The area to the left is straightforward, since it's in the lower tail (calc in R):

 > pf(r,28,34) 
 [1] 0.2210767

We need the probability for the other tail. If the distribution was symmetric in the inverse, it would be straightforward to do this.

A common convention with the ratio of variances F-test (which is similarly two tailed) is simply to double the one-tailed p-value (effectively what is going on as here; that's also what seems to be done in R, for example); in this case it gives a p-value of 0.44.

However, if you do it with a formal rejection rule, by putting an area of $\alpha/2$ in each tail, you'd get critical values as described here. The p-value is then the largest $\alpha$ that would lead to rejection, which is equivalent to adding the one tailed p-value above to the one-tailed p-value in the other tail for the degrees of freedom interchanged. In the above example that gives a p-value of 0.43.

$\endgroup$
  • $\begingroup$ I'm guessing this is just me being thick, but where does 0.7515 come from? $\endgroup$ – Jonathan Dobbie Nov 20 '13 at 21:00
  • $\begingroup$ r = mean(x)/mean(y) = 0.7515 - that is, "The ratio of means" $\endgroup$ – Glen_b Nov 20 '13 at 21:48
  • $\begingroup$ Okay, awesome. I got 0.67, but that is probably just due to a data entry error. $\endgroup$ – Jonathan Dobbie Nov 20 '13 at 23:11
  • 1
    $\begingroup$ I have made the distinction between the population means and the resulting sample means more clear $\endgroup$ – Glen_b Nov 21 '13 at 5:12
  • $\begingroup$ (+1) But though it's tangential, I don't understand the last paragraph. How is doubling the one-tailed p-value not equivalent to finding the largest $\alpha$, with an area $\frac{\alpha}{2}$ in each tail, that would lead to rejection? Why would you interchange the degrees of freedom at all? $\endgroup$ – Scortchi - Reinstate Monica Mar 22 '16 at 12:50
3
$\begingroup$

As an addendum to @Glen_b's answer, the likelihood ratio is $$n_x\log \frac{n_x}{\sum x_i} +n_y \log \frac{n_y}{\sum y_j} -(n_x+n_y)\log\frac{n_x+n_y}{\sum x_i +\sum y_j}$$ which you can reärrange to $$n_x\log\left(\frac{n_x}{n_y} + \frac{1}{r}\right) + n_y\log\left(\frac{n_y}{n_x}+r\right) + n_x\log\frac{n_y}{n_x+n_y} + n_y\log \frac{n_x}{n_x+n_y}$$ where $r=\frac{\bar{x}}{\bar{y}}$. There's a single minimum at $r=1$, so the F-test is indeed the likelihood ratio test against one-sided alternatives to the null hypothesis of identical distributions.

To perform the likelihood-ratio test proper for a two-sided alternative you can still use the F-distribution; you simply need to find the other value of the ratio of sample means $r_\mathrm{ELR}$ for which the likelihood ratio is equal to that of the observed ratio $r_\mathrm{obs}$, & then $\Pr(R>r_\mathrm{ELR})$. For this example $r_\mathrm{ELR}=1.3272$, & $\Pr(R>r_\mathrm{ELR})=0.2142$, giving an overall p-value of $0.4352$, (rather close to that obtained by the chi-square approximation to the distribution of twice the log likelihood ratio, $0.4315$).

enter image description here

But doubling the one-tailed p-value is perhaps the most common way to obtain a two-tailed p-value: it's equivalent to finding the value of the ratio of sample means $r_\mathrm{ETP}$ for which the tail probability $\Pr(R>r_\mathrm{ETP})$ is equal to $\Pr(R<r_\mathrm{obs})$, & then finding $\Pr(R>r_\mathrm{ETP})$. Explained like that, it might seem to be putting the cart before the horse in letting tail probabilities define the extremeness of a test statistic, but it can be justified as being in effect two one-tailed tests (each the LRT) with a multiple comparisons correction—& people are usually interested in claiming either that $\mu_x > \mu_y$ or that $\mu_x < \mu_y$ rather than that either $\mu_x > \mu_y$ or $\mu_x < \mu_y$. It's also less fuss, & even for fairly small sample sizes, gives much the same answer as the two-tailed LRT proper.

enter image description here

R code follows:

x <- c(12.173, 3.148, 33.873, 0.160, 3.054, 11.579, 13.491, 7.048, 48.836,
       16.478, 3.323, 3.520, 7.113, 5.358)

y <- c(7.635, 1.508, 29.987, 13.636, 8.709, 13.132, 12.141, 5.280, 23.447, 
       18.687, 13.055, 47.747, 0.334,7.745, 26.287, 34.390, 9.596)

# observed ratio of sample means
r.obs <- mean(x)/mean(y)

# sample sizes
n.x <- length(x)
n.y <- length(y)

# define log likelihood ratio function
calc.llr <- function(r,n.x,n.y){
  n.x * log(n.x/n.y + 1/r) + n.y*log(n.y/n.x + r) + n.x*log(n.y/(n.x+n.y)) + n.y*log(n.x/(n.x+n.y))
}

# observed log likelihood ratio
calc.llr(r.obs,n.x, n.y) -> llr.obs

# p-value in lower tail
pf(r.obs,2*n.x,2*n.y) -> p.lo

# find the other ratio of sample means giving an LLR equal to that observed
uniroot(function(x) calc.llr(x,n.x,n.y)-llr.obs, lower=1.2, upper=1.4, tol=1e-6)$root -> r.hi

#p.value in upper tail
p.hi <- 1-pf(r.hi,2*n.x,2*n.y)

# overall p.value
p.value <- p.lo + p.hi

#approximate p.value
1-pchisq(2*llr.obs, 1)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.