1
$\begingroup$

I am using the twang package to estimate the propensity scores of participants in two active labour market programmes. One of them is public works (with 20 000 participants) and the other is voluntary work (13 000), and the participants in both programmes differ in some covariates. My ultimate research question is: How would public works participants have fared in terms of exit to the labour market if they were enrolled to the voluntary programme? (“ATT -- average treatment effect in the treated population” design)

Before I can answer the research question I need to assess the quality of my propensity scores to which the twang package offers several diagnostic tools. One of them is in the balance.table that provides both the standard effect size and p-value of t-test (or chi-square for categorical variables) before and after weighting the covariates. One of the things I do not quite understand is what the t-test refers to in my case if I use administrative data and not a sample.

After weighting, the standard effect sizes for all of my covariates have fallen below 0.2, but I still have low p-values. The distribution of p-values looks like this: enter image description here

Do I interpret the situation well if I say that the difference between my two groups is small (measured by mean of covariates) but they remain statistically significant. I.e.: If we were to choose a random sample from the control group, it would probably be different from my weighted sample?

If my interpretation is correct then is this something to worry about in ps score weighting, or it is just a matter of having large n, where even a small difference can be significant?

I hope I managed to explain myself clearly. Please, let me know if you need any more information.

Thank you in advance.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

This is a problem with using hypothesis tests to assess balance. This is an inappropriate practice according to many researchers (e.g., Stuart, 2010; Ho, Imai, King, & Stuart, 2007; Ali et al., 2015). In particular, it is inappropriate for the reasons you mention: balance is entirely conflated with sample size because sample size is used to compute the balance statistic when the balance statistic is a p-value, but sample size is actually irrelevant to balance. Rely instead on the standardized mean difference, variance ratios, and the KS-statistic (not p-value) to assess balance in your data.

I wrote an R package, cobalt, which is designed to assess balance using only the methods recommended in the literature, and it is fully compatible with output objects from twang.

Improve this answer
$\endgroup$
5
  • $\begingroup$ are you saying that I should stop looking at any p-values in the balance.table, including KS p-values? That actually would be quite a relieve becuase the plot on my KS p-values looks even weirder than the one I posted above (I have a bunch 1s for unweighted covariates that actually decrease after weighting). Thanks for the cobalt hint, I will study it! $\endgroup$
    – malasi
    Jun 12, 2018 at 9:00
  • $\begingroup$ I would fit a saturated propensity score model, then you can almost ignore balance because balance is guaranteed unless there are interactions in the PS. An example of a saturated model is an additive model in which a flexible spline function is used for each continuous predictor. $\endgroup$
    – Frank Harrell
    Jun 12, 2018 at 11:24
  • $\begingroup$ @malasi, you can ignore p-values, but if they point to a legitimate imbalance then you still may have work to do. If your KS-statistics are high, you still have imbalance, regardless of the associated p-value. Low p-values tend to follow large KS-statistics and large SMDs, but you can assess those directly. Feel free to email me if you have questions about cobalt. Also, with such a large sample, you should seek better balance than 0.2 SMD and ensure balance on interactions as well. $\endgroup$
    – Noah
    Jun 12, 2018 at 14:24
  • $\begingroup$ @FrankHarrell, that sounds like an interesting idea. Do you have a link to a tutorial with some R or SAS code to implement such a model? $\endgroup$
    – Noah
    Jun 12, 2018 at 14:25
  • $\begingroup$ My RMS book and course notes. $\endgroup$
    – Frank Harrell
    Jun 13, 2018 at 11:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.