I'm still new to R and most probably this is a rookie question, but maybe some of you could help me understand what is happening.

I'm analyzing some results of an experiment in which I have three treatments and three outcomes. For simplicity I will only take the control and treatment 1 and the three outcome measures.

Before running any statistical test, I needed to create a weighting variable because the sample is not representative of the population. To do so, I've used the anesrake package. This is the code of the weighting:

Sex <- c(.49,.51)
agecat  <- c(.085, .136, .184, .194, .401)
names(Sex) <- c("Male", "Female")
names(agecat) <- c("18-24", "25-34","35-44","45-54", "Mas que 55")
targets <- list(Sex, agecat)
names(targets) <- c("Sex", "agecat")
#I create a unique id inmy main dataframe
main_data_id$caseid <- 1:length(main_data_id$Sex)

main_data_id$Sex <- as.factor(main_data_id$Sex) 
main_data_id$agecat <- as.factor(main_data_id$agecat)
main_data_id$education <- as.factor(main_data_id$education)

#I check the difference between the population and sample distribution of
anesrakefinder(targets, main_data_id, choosemethod = "total") #all greater than 5%points

main_data_id$caseid <- as.numeric(main_data_id$caseid)

weighted_data <- anesrake(targets, main_data_id, caseid = main_data_id$caseid,
                    verbose= FALSE, cap = 5, choosemethod = "total",
                    type = "pctlim", pctlim = .05 , nlim = 5,
                    iterate = TRUE , force1 = TRUE)

# add weights to the dataset
main_data_id$weightvec  <- unlist(weighted_data[1])
n  <- length(main_data_id$Sex)

After having created the weighting variable I'm now performing t-tests (weighted and unweighted) to compare the mean of measure 1/2/3 between control and treatment.

When comparing the results of the weighted and unweighted t-tests I've noticed something unusual. The treatment was not significant on measure 2 and 3 and the results resulting from weighted and unweighted t-tests are very similar. EX:


    Welch Two Sample t-test

data:  treatment1_2$Measure1 by treatment1_2$treatment
t = -0.62172, df = 509.92, p-value = 0.5344
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.4920101  0.2554664
sample estimates:
mean in group 0 mean in group 1 
       7.320463        7.438735 


    t.value          df     p.value 
  0.8140750 505.6230093   0.4159852 

Difference     Mean.x     Mean.y   Std. Err 
 0.1553317  7.3958085  7.2404768  0.1908076

But on measure 1 it was significant when performing the normal t-test, but when doing the weighted one the p-value is completely off. Example:


    Welch Two Sample t-test

data:  treatment2_3$Measure1 by treatment2_3$treatment
t = 3.5345, df = 508.84, p-value = 0.0004458
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.308046 1.079077
sample estimates:
mean in group 0 mean in group 1 
       7.438735        6.745174 


[1] "Two Sample Weighted T-Test (Welch)"

     t.value           df      p.value 
4.133672e+00 4.976646e+02 4.189322e-05 

Difference     Mean.x     Mean.y   Std. Err 
 0.8189135  7.2404768  6.4215632  0.1981080 

The same thing happens across all treatment groups and outcome measures. When the treatment is not significant in the unweighted test, then the weighted t-test has a similar result. When the treatment is significant, there is a huge difference between weighted and unweighted t-tests.

I've also run some weighted and unweighted regressions and the same thing happens (however, in the weighted regression, when the treatment is significant and the p-value is high it still gets marked with three stars).

Do you know why this is happening? Also, do you need a reproducible example? Thank you so much for any help!!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.