I'm still new to R and most probably this is a rookie question, but maybe some of you could help me understand what is happening.
I'm analyzing some results of an experiment in which I have three treatments and three outcomes. For simplicity I will only take the control and treatment 1 and the three outcome measures.
Before running any statistical test, I needed to create a weighting variable because the sample is not representative of the population. To do so, I've used the anesrake package. This is the code of the weighting:
Sex <- c(.49,.51) agecat <- c(.085, .136, .184, .194, .401) names(Sex) <- c("Male", "Female") names(agecat) <- c("18-24", "25-34","35-44","45-54", "Mas que 55") targets <- list(Sex, agecat) names(targets) <- c("Sex", "agecat") #I create a unique id inmy main dataframe main_data_id$caseid <- 1:length(main_data_id$Sex) main_data_id$Sex <- as.factor(main_data_id$Sex) main_data_id$agecat <- as.factor(main_data_id$agecat) main_data_id$education <- as.factor(main_data_id$education) #I check the difference between the population and sample distribution of anesrakefinder(targets, main_data_id, choosemethod = "total") #all greater than 5%points main_data_id$caseid <- as.numeric(main_data_id$caseid) weighted_data <- anesrake(targets, main_data_id, caseid = main_data_id$caseid, verbose= FALSE, cap = 5, choosemethod = "total", type = "pctlim", pctlim = .05 , nlim = 5, iterate = TRUE , force1 = TRUE) summary(weighted_data) # add weights to the dataset main_data_id$weightvec <- unlist(weighted_data) n <- length(main_data_id$Sex)
After having created the weighting variable I'm now performing t-tests (weighted and unweighted) to compare the mean of measure 1/2/3 between control and treatment.
When comparing the results of the weighted and unweighted t-tests I've noticed something unusual. The treatment was not significant on measure 2 and 3 and the results resulting from weighted and unweighted t-tests are very similar. EX:
Welch Two Sample t-test data: treatment1_2$Measure1 by treatment1_2$treatment t = -0.62172, df = 509.92, p-value = 0.5344 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.4920101 0.2554664 sample estimates: mean in group 0 mean in group 1 7.320463 7.438735
$coefficients t.value df p.value 0.8140750 505.6230093 0.4159852 $additional Difference Mean.x Mean.y Std. Err 0.1553317 7.3958085 7.2404768 0.1908076
But on measure 1 it was significant when performing the normal t-test, but when doing the weighted one the p-value is completely off. Example:
Welch Two Sample t-test data: treatment2_3$Measure1 by treatment2_3$treatment t = 3.5345, df = 508.84, p-value = 0.0004458 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: 0.308046 1.079077 sample estimates: mean in group 0 mean in group 1 7.438735 6.745174
 "Two Sample Weighted T-Test (Welch)" $coefficients t.value df p.value 4.133672e+00 4.976646e+02 4.189322e-05 $additional Difference Mean.x Mean.y Std. Err 0.8189135 7.2404768 6.4215632 0.1981080
The same thing happens across all treatment groups and outcome measures. When the treatment is not significant in the unweighted test, then the weighted t-test has a similar result. When the treatment is significant, there is a huge difference between weighted and unweighted t-tests.
I've also run some weighted and unweighted regressions and the same thing happens (however, in the weighted regression, when the treatment is significant and the p-value is high it still gets marked with three stars).
Do you know why this is happening? Also, do you need a reproducible example? Thank you so much for any help!!