2
$\begingroup$

I noticed that in the "General Procedures" for Propensity Score mentioned in its Wikipedia entry, it begins:

  1. Run logistic regression:

    Dependent variable: Z = 1, if unit participated (i.e. is member of the treatment group); Z = 0, if unit did not participate (i.e. is member of the control group). Choose appropriate confounders (variables hypothesized to be associated with both treatment and outcome) Obtain an estimation for the propensity score: predicted probability (p) or log[p/(1 − p)].

  2. Check that propensity score is balanced across treatment and comparison groups, and check that covariates are balanced across treatment and comparison groups within strata of the propensity score.

Then their 3rd step proceeds to matching. To me the first part of #2 seems flat wrong. Why should your propensity scores be balanced between Treatment and Control before matching?

If you have perfectly balanced propensity scores in the Treatment and Control groups without matching / weighting, doesn't that mean that either:

  1. You have a randomized experiment or a situation effectively the same as one - congratulations, you do not need to do Propensity Score Matching, or
  2. You've failed include relevant confounders (observable characteristics which are different between Treatment and Control and which are correlated with the outcome), thus your Propensity Score model failed and your PSM will have no effect in reducing selection bias.

I'd find it much more comforting if the propensity scores were statistically significantly different between Treatment and Control in an observational study where I know that assignment to treatment is non-random, because that would at least imply that the score can be used to control for confounding variation that I might know exists. Am I correct or am I missing something?

Improve this question
$\endgroup$
1
  • $\begingroup$ I've submitted a correction to Wikipedia, but without logging in so the change probably won't be federated anytime soon, if at all. Let me know if I was wrong to do so. $\endgroup$
    – Hack-R
    Oct 22, 2020 at 23:04

1 Answer 1

Reset to default
0
$\begingroup$

I agree that this an unnecessary step, but it's not an egregious error. It is a good idea to take a look at the initial imbalance in your data before matching so you can see for which variables special attention may need to be paid in matching. I would only change the word "that" to "if" in that step, which is a very minor change. If the propensity score and covariates are well balanced before matching, then, yes, matching may not be necessary.

Improve this answer
$\endgroup$
5
  • $\begingroup$ "It is a good idea to take a look at the initial imbalance in your data before matching so you can see for which variables special attention may need " - the first part of #2 which is what's in question is referring to the treatment assignment (Z) not to covariates (X), so I think is actually is an egregious error. There should be imbalance in the the treatment assignment vector Z, if the propensity score was constructed with relevant confounders X. $\endgroup$
    – Hack-R
    Oct 23, 2020 at 15:52
  • $\begingroup$ #2 doesn't mention the treatment assignment vector... it mentions the propensity score, which is a function of the observed covariates. There should only be imbalance in the propensity score if the measured covariates cause or predict treatment assignment; but that may not be the case, and whether they do or not does not imply the presence or absence of unobserved confounding. If there isn't imbalance in the propensity score, then the selection process might be random, which is a good thing. $\endgroup$
    – Noah
    Oct 23, 2020 at 18:33
  • $\begingroup$ Similarly, imbalance in the propensity score doesn't mean you've captured relevant confounders. It just means the covariates you observed are associated with treatment assignment. You shouldn't be "comforted" either way. Imbalance in the propensity score is just a proxy for imbalance in the covariates that were used to construct it; it doesn't tell you anything special about confounding beyond the balance of the covariates you observed. $\endgroup$
    – Noah
    Oct 23, 2020 at 18:37
  • $\begingroup$ "Check that propensity score is balanced across treatment and comparison groups and...". That's the part I'm referring to. If it were just the covariates I'd have no problem except that they used the term "balance" (which in PSM language is referring to Z, as opposed to "overlap" for X), but everything before "and" seems to refer to Z. If they had said "Check that propensity score is balanced across treatment and comparison groups for covariates" that would be totally different, but the and implies that "Check .. across T and comparison groups" is a set of instructions referring to Z. $\endgroup$
    – Hack-R
    Oct 23, 2020 at 18:40
  • $\begingroup$ Balance for a variable $V$ means $f(V|Z=1) \approx f(V|Z=0)$, where $f(V|Z=1)$ is the distribution of $V$ in the treated group and $f(V|Z=0)$ is the distribution of $V$ in the comparison group. Balance on the propensity score, $e(X) = P(Z=1|X)$, simply is to replace $V$ with $e(X)$. It seems like you think the propensity score is related only to $Z$, but it's just a unidimensional summary of $X$. Balance on the propensity score is a summary of balance on the covariates that are used to form the propensity score. $\endgroup$
    – Noah
    Oct 24, 2020 at 8:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.