I am using the twang
package to estimate the propensity scores of participants in two active labour market programmes. One of them is public works (with 20 000 participants) and the other is voluntary work (13 000), and the participants in both programmes differ in some covariates. My ultimate research question is: How would public works participants have fared in terms of exit to the labour market if they were enrolled to the voluntary programme? (“ATT -- average treatment effect in the treated population” design)
Before I can answer the research question I need to assess the quality of my propensity scores to which the twang
package offers several diagnostic tools. One of them is in the balance.table
that provides both the standard effect size and p-value of t-test (or chi-square for categorical variables) before and after weighting the covariates. One of the things I do not quite understand is what the t-test refers to in my case if I use administrative data and not a sample.
After weighting, the standard effect sizes for all of my covariates have fallen below 0.2, but I still have low p-values. The distribution of p-values looks like this:
Do I interpret the situation well if I say that the difference between my two groups is small (measured by mean of covariates) but they remain statistically significant. I.e.: If we were to choose a random sample from the control group, it would probably be different from my weighted sample?
If my interpretation is correct then is this something to worry about in ps score weighting, or it is just a matter of having large n, where even a small difference can be significant?
I hope I managed to explain myself clearly. Please, let me know if you need any more information.
Thank you in advance.