# Unusual p-values after weighting

I'm still new to R and most probably this is a rookie question, but maybe some of you could help me understand what is happening.

I'm analyzing some results of an experiment in which I have three treatments and three outcomes. For simplicity I will only take the control and treatment 1 and the three outcome measures.

Before running any statistical test, I needed to create a weighting variable because the sample is not representative of the population. To do so, I've used the anesrake package. This is the code of the weighting:

Sex <- c(.49,.51)
agecat  <- c(.085, .136, .184, .194, .401)
names(Sex) <- c("Male", "Female")
names(agecat) <- c("18-24", "25-34","35-44","45-54", "Mas que 55")
targets <- list(Sex, agecat)
names(targets) <- c("Sex", "agecat")
#I create a unique id inmy main dataframe
main_data_id$$caseid <- 1:length(main_data_id$$Sex)

main_data_id$$Sex <- as.factor(main_data_id$$Sex)
main_data_id$$agecat <- as.factor(main_data_id$$agecat)
main_data_id$$education <- as.factor(main_data_id$$education)

#I check the difference between the population and sample distribution of
anesrakefinder(targets, main_data_id, choosemethod = "total") #all greater than 5%points

main_data_id$$caseid <- as.numeric(main_data_id$$caseid)

weighted_data <- anesrake(targets, main_data_id, caseid = main_data_id$caseid, verbose= FALSE, cap = 5, choosemethod = "total", type = "pctlim", pctlim = .05 , nlim = 5, iterate = TRUE , force1 = TRUE) summary(weighted_data) # add weights to the dataset main_data_id$$weightvec <- unlist(weighted_data[1]) n <- length(main_data_id$$Sex)  After having created the weighting variable I'm now performing t-tests (weighted and unweighted) to compare the mean of measure 1/2/3 between control and treatment. When comparing the results of the weighted and unweighted t-tests I've noticed something unusual. The treatment was not significant on measure 2 and 3 and the results resulting from weighted and unweighted t-tests are very similar. EX: Unweighted:  Welch Two Sample t-test data: treatment1_2$$Measure1 by treatment1_2$$treatment t = -0.62172, df = 509.92, p-value = 0.5344 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.4920101 0.2554664 sample estimates: mean in group 0 mean in group 1 7.320463 7.438735  Weighted: $coefficients
t.value          df     p.value
0.8140750 505.6230093   0.4159852

$additional Difference Mean.x Mean.y Std. Err 0.1553317 7.3958085 7.2404768 0.1908076  But on measure 1 it was significant when performing the normal t-test, but when doing the weighted one the p-value is completely off. Example: Unweighted:  Welch Two Sample t-test data: treatment2_3$$Measure1 by treatment2_3$$treatment t = 3.5345, df = 508.84, p-value = 0.0004458 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: 0.308046 1.079077 sample estimates: mean in group 0 mean in group 1 7.438735 6.745174  Weighted: [1] "Two Sample Weighted T-Test (Welch)"$coefficients
t.value           df      p.value
4.133672e+00 4.976646e+02 4.189322e-05