# Low p-values in propensity scores

I am using the twang package to estimate the propensity scores of participants in two active labour market programmes. One of them is public works (with 20 000 participants) and the other is voluntary work (13 000), and the participants in both programmes differ in some covariates. My ultimate research question is: How would public works participants have fared in terms of exit to the labour market if they were enrolled to the voluntary programme? (“ATT -- average treatment effect in the treated population” design)

Before I can answer the research question I need to assess the quality of my propensity scores to which the twang package offers several diagnostic tools. One of them is in the balance.table that provides both the standard effect size and p-value of t-test (or chi-square for categorical variables) before and after weighting the covariates. One of the things I do not quite understand is what the t-test refers to in my case if I use administrative data and not a sample.

After weighting, the standard effect sizes for all of my covariates have fallen below 0.2, but I still have low p-values. The distribution of p-values looks like this:

Do I interpret the situation well if I say that the difference between my two groups is small (measured by mean of covariates) but they remain statistically significant. I.e.: If we were to choose a random sample from the control group, it would probably be different from my weighted sample?

If my interpretation is correct then is this something to worry about in ps score weighting, or it is just a matter of having large n, where even a small difference can be significant?

I hope I managed to explain myself clearly. Please, let me know if you need any more information.

I wrote an R package, cobalt, which is designed to assess balance using only the methods recommended in the literature, and it is fully compatible with output objects from twang.
• are you saying that I should stop looking at any p-values in the balance.table, including KS p-values? That actually would be quite a relieve becuase the plot on my KS p-values looks even weirder than the one I posted above (I have a bunch 1s for unweighted covariates that actually decrease after weighting). Thanks for the cobalt hint, I will study it! – malasi Jun 12 '18 at 9:00
• @malasi, you can ignore p-values, but if they point to a legitimate imbalance then you still may have work to do. If your KS-statistics are high, you still have imbalance, regardless of the associated p-value. Low p-values tend to follow large KS-statistics and large SMDs, but you can assess those directly. Feel free to email me if you have questions about cobalt. Also, with such a large sample, you should seek better balance than 0.2 SMD and ensure balance on interactions as well. – Noah Jun 12 '18 at 14:24