1
$\begingroup$

I am currently working with a dataset that comprises measurements of multiple participants on two variables. I'd like to test the mean difference between both variables on statistical significance.

However, some observations on both variables are from the same subjects, whereas others aren't (i.e. one might also think of the dataset as a repeated measurement with missing data). Thus, simple t-test doesn't seem to apply, given that observations are neither "fully dependent" nor "fully independent" (of course one might exclude those participants with only one observation, but so far, I’d like to refrain from that).

I thought that maybe I could drive some bootstrap approach, by simply bootstrapping the t-statistic for independent observations under the nullhypotheses. However, this approach does seem to produce a type I error above 5%.

Here is my R code for the simulation:

rm(list = ls())
set.seed(123)
library(MASS)


mu1 = 0 # sampling under the null hypotheses
mu2 = 0
mu_vec = c(mu1,mu2) # (for the dependent measurements)
sig = 2 # some arbitrary variance
SIG = matrix(c(2,-1,-1,2), c(2,2)) # and covariance-matrix
n1 = 7 # number of independent samples in the first vector
n2 = 4 # .. in the second vector


P = 5000
ps = vector(length = P)
for(j in 1:P){

  paired = rbind(mvrnorm(1,mu_vec, SIG),
                 mvrnorm(1,mu_vec, SIG),
                 mvrnorm(1,mu_vec, SIG),
                 mvrnorm(1,mu_vec, SIG),
                 mvrnorm(1,mu_vec, SIG),
                 mvrnorm(1,mu_vec, SIG),
                 mvrnorm(1,mu_vec, SIG)) # simulating the data for the dependent measurements
  random_x1 = c(paired[,1], rnorm(n1, mu1, sig)) # combining them 
  random_x2 = c(paired[,2], rnorm(n2, mu2, sig)) #  with simulated data for the independent measurements
  t_emp = t.test(random_x1, random_x2, var.equal = TRUE) # our empirical t-value (for independent measurements)
  t_emp = t_emp$statistic

  #The Bootstrap approach
  random_x1 = random_x1 - mean(random_x1) 
  random_x2 = random_x2 - mean(random_x2)
  B      <- 1000
  t.vect <- vector(length=B) # storing the bootstrapped t-values
    for(i in 1:B){
      boot.x1 <- sample(random_x1, size=length(random_x1), replace=T) # resampling
      boot.x2 <- sample(random_x2, size=length(random_x2), replace=T) # resampling
      test = t.test(boot.x1, boot.x2, var.equal = TRUE) 
      t.vect[i] <- test$statistic
    }

  ps[j] = sum( abs(t_emp)  <= t.vect | -abs(t_emp) >= t.vect)/B # calculating the empirical p value 
  # i.e. how many values of the bootstrapped t-distribution are above or below our empirical t-value?
}

type1 = sum(ps <= 0.05)/P # how many times would we reject the null hypothesis?
type1
[1] 0.07

My Questions:

1.) Is there a better resampling method to handle this situation?

2.) Or does there even exist a common statistical test for these kinds of samples?

$\endgroup$
  • $\begingroup$ "[S]ome observations on both variables are from the same subjects, whereas others aren't ." If -none_ of the subjects provided both measurements, would the data be completely useless? If not, what would you do? Could you use a 2-sample test? $\endgroup$ – BruceET Aug 26 '19 at 8:57
  • $\begingroup$ "If -none- of the subjects provided both measurements, would the data be completely useless?". No, around 70% of the participants have measurements on both variables. I'm not sure if I can take a 2-sample t-test; testing the difference D = X1 - X2 is not possible due to the missing values and a t-test for independent measurements is also not applicable because - at least- some measurements are dependent. Of course, I could delete all missing values and take a simple t-test on D, however in my case this would limit the value of the study. $\endgroup$ – bucky Aug 27 '19 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.