2
$\begingroup$

I aim to check the robustness of 2 groups t-test when samples come from lightly skewed distributions. To approach the problem I though about performing a Montecarlo based robustness analysis using experimental sample sizes and zero centered means, but, for setting the expected skewnes and variance of each group populations I wonder if it would be right to resampling from sample distributions (bootstrapping strategy) instead of sampling from a expected distribution of the population according to the nature of the variable studied.

An example on sleep R dataset:

plot(extra ~ group, data = sleep)

enter image description here

robust_fun<-function(x,y,iter=10^5, alter="two.sided", welch=F)
{
  # Robustness of t.test. Bootstrap from empirical distributions.
  
  # sample sizes
  nx<-length(as.numeric(x)) 
  ny<-length(as.numeric(y))
  
  # dsitribution centered in the mean
  x<-x-mean(x,na.rm=T)
  y<-y-mean(y,na.rm=T)
  
  # resampling with replazment
  pval<-replicate(iter, t.test(remp(nx,x),remp(ny,y), var.equal=welch, alternative=alter)$p.val)
  mean(pval <= 0.05)
}

sleep_data<-split(sleep$extra,sleep$group)
robust_fun(sleep_data$`1`,sleep_data$`2`,10000)

For 0.05 significance level the result was 0.0521. Is this robustneess analysis trustworthy?

Improve this question
$\endgroup$
5
  • $\begingroup$ It's hard to tell what you're trying to achieve with this analysis. You're forcing the mean of x and y to be 0, and then resampling each with replacement (assuming this is what remp does) and testing the difference between them (which by definition is now 0) using a Student's t-test. Having ~5% of p-values <= .05 is exactly what we'd expect here. If you want a test of the difference that's robust to data from lightly skewed distributions, why not use the regular Welch test and skip the bootstrap? Unless you want something different? $\endgroup$
    – awhug
    Commented May 16, 2022 at 0:29
  • $\begingroup$ @awhug I'll try to clarify the goal. Because the t-test requires that the sample come from normal distributions, the percentage of p-values < 0.05 does not have to be exactly 0.05 in the example. When performing a t-test on highly skewed distributions robustness is lost, meaning that the empirical type I error is not equal to the nominal alpha (0.05). Various threads discuss the issue. $\endgroup$
    – Tomás Navarro
    Commented May 16, 2022 at 10:05
  • $\begingroup$ Here the top answer explains a Monte Carlo method to check robustness: stats.stackexchange.com/questions/242037/… I propose instead to use the empirical distribution of the input samples (which exactly remp fun does). I can switch from t-test to Welch's by simply setting the Welch parameter to TRUE, but the question is the same, is this bootstrapping approach correct? $\endgroup$
    – Tomás Navarro
    Commented May 16, 2022 at 10:07
  • $\begingroup$ Thanks that helps for context. In the linked answer, the data generating process was known, so it was easy to verify theoretical vs. empirical false positive rates of the t-test through simulation under a variety of known sampling distributions. Given you've only presented one dataset with two groups drawn from potentially unknown/unknowable distributions, it's still a bit unclear what you're after. Are you ultimately aiming to make some conclusion about the t-test itself, or the results of a t-test as applied to a particular sample? What do you propose your analysis tells you? $\endgroup$
    – awhug
    Commented May 17, 2022 at 0:22
  • $\begingroup$ Thank you for your interest. My aim is checking if t-test aplied to these particular samples from potentially unknown distributions (as you pointed out) is robust enough to be applied after to the original samples (no 0 centered). The question could be Can I trust t-test result when applied to these samples? Due to original population distributions are unknown the main assumption here (big one) is that sample distributions emulate population ones, as an alternative to "infer" the expected population distribution from the nature of the variable studied or previous information. $\endgroup$
    – Tomás Navarro
    Commented May 17, 2022 at 9:00

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.