1
$\begingroup$

Is it possible to run a hypothesis test on two means in a group when only given the means and no raw data?

For example I have and average over 30 days for both groups- but no data to form a variance on it.

$\endgroup$
4
  • $\begingroup$ Do you have a standard deviation or any other information about the distributions, or do you just have to determine if some number (mean #1) is significantly different from some other number (mean #2)? $\endgroup$
    – Dave
    Aug 2 at 21:25
  • 2
    $\begingroup$ What is your hypothesis? Some are testable (under mild assumptions) and others are not. $\endgroup$
    – whuber
    Aug 2 at 21:27
  • $\begingroup$ Just testing whether one mean is significantly different from another $\endgroup$
    – Tyler
    Aug 2 at 22:25
  • 1
    $\begingroup$ Even if you did have standard deviation info, you likely have non-independence, given the time series data. So the usual types of t- or z-tests would not be valid. $\endgroup$ Aug 3 at 11:12
0
$\begingroup$

The time series nature of your data means that your observations lack independence. There are ways to deal with time series data, but the usual hypothesis tests like a t-test assume independence. When independence of observations is violated, there is weird behavior. The type I error rate is too high for a true null hypothesis, and the type II error rate is too high for a true null hypothesis. I will give a simulation and graphs showing this.

library(ggplot2)
library(MASS)
set.seed(2021)
N <- 30
B <- 1000
p_ind_ho <- p_dep_ho <- p_ind_ha <- p_dep_ha <- rep(NA, B)
for (i in 1:B){
  
  # Simulate data where mu0 = mu1 = 0 and observations are independent
  #
  x <- rnorm(N)
  y <- rnorm(N)
  
  # Run the t-test, store p-value
  #
  p_ind_ho[i] <- t.test(x, y)$p.value
  
  # Simulate data where mu0 = mu1 and observations are dependent
  #
  x <- c(MASS::mvrnorm(N/2, c(0, 0), matrix(c(1, 0.8, 0.8, 1), 2, 2)))
  y <- c(MASS::mvrnorm(N/2, c(0, 0), matrix(c(1, 0.8, 0.8, 1), 2, 2)))
  
  # Run the t-test, store p-value
  #
  p_dep_ho[i] <- t.test(x, y)$p.value
  
  # Simulate data where mu0 =/= mu1 and observations are independent
  #
  x <- rnorm(N)
  y <- rnorm(N, 0.5)
  
  # Run the t-test, store p-value
  #
  p_ind_ha[i] <- t.test(x, y)$p.value
  
  # Simulate data where mu0 =/= mu1 and observations are dependent
  #
  x <- c(MASS::mvrnorm(N/2, c(0, 0), matrix(c(1, 0.8, 0.8, 1), 2, 2)))
  y <- c(MASS::mvrnorm(N/2, c(0.25, 0.25), matrix(c(1, 0.8, 0.8, 1), 2, 2)))
  
  # Run the t-test, store p-value
  #
  p_dep_ha[i] <- t.test(x, y)$p.value    
  
}

s <- seq(0, 1, 0.001)
d0 <- data.frame(x = s, cdf = ecdf(p_ind_ho)(s), null = "True Null", dependence = "Independent")
d1 <- data.frame(x = s, cdf = ecdf(p_dep_ho)(s), null = "True Null", dependence = "Autocorrelation")
d2 <- data.frame(x = s, cdf = ecdf(p_ind_ha)(s), null = "False Null", dependence = "Independent")
d3 <- data.frame(x = s, cdf = ecdf(p_dep_ha)(s), null = "False Null", dependence = "Autocorrelation")
d <- rbind(d0, d1, d2, d3)
ggplot(d, aes(x = x, y = cdf, col = dependence)) +
  geom_line() +
  geom_point() +
  geom_abline(intercept = 0, slope = 1) +
  facet_grid(~null) +
  theme_bw()

enter image description here On the left, we see that, when the null is true, the independent data give a uniform distribution of p-values, but the dependent data give a distribution skewed towards rejection, which is undesirable. The test is too powerful.

On the right, we see that, when the null is false, the independent data give a distribution of p-values more skewed towards rejection than the dependent data. The test with independent data is more powerful.

Thus, the dependent data give too many type I errors and too many type II errors, the worst of both worlds!

(Zooming in to $\alpha$ levels of interest with a command like xlim(0, 0.1) does not change the story, and I have played with this with other tests like Wilcoxon, only to get the same result that both error rates are too high.)

$\endgroup$
5
  • $\begingroup$ Re "the distance between the means is meaningless, for hypothesis testing, without some sense of the variation." One would think so, but it's not true (surprisingly). For instance, it's possible to construct a 95% confidence interval for the difference assuming only that the sampling distributions of the means are the same and are Normal. $\endgroup$
    – whuber
    Aug 3 at 12:33
  • $\begingroup$ @whuber Do you not make some assumption about the variance? $\endgroup$
    – Dave
    Aug 3 at 12:48
  • 1
    $\begingroup$ No -- that's what's surprising. See Carlos Rodriguez, Confidence Intervals from One Observation. Apply it to the difference between the two means. $\endgroup$
    – whuber
    Aug 3 at 13:26
  • $\begingroup$ @whuber Weird! And I've edited that section out of my post (though the issue with autocorrelation remains). $\endgroup$
    – Dave
    Aug 3 at 13:52
  • $\begingroup$ I don't see any issue with autocorrelation. Its presence will change how one estimates the sampling variance of the mean and it will change the d.f. in a t-test, but it doesn't make it any less possible to compare the two means. What you have demonstrated is that in the presence of autocorrelation it would be an error to apply the usual t-test. $\endgroup$
    – whuber
    Aug 3 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.