I noticed that in the "General Procedures" for Propensity Score mentioned in its Wikipedia entry, it begins:
Run logistic regression:
Dependent variable: Z = 1, if unit participated (i.e. is member of the treatment group); Z = 0, if unit did not participate (i.e. is member of the control group). Choose appropriate confounders (variables hypothesized to be associated with both treatment and outcome) Obtain an estimation for the propensity score: predicted probability (p) or log[p/(1 − p)].
Check that propensity score is balanced across treatment and comparison groups, and check that covariates are balanced across treatment and comparison groups within strata of the propensity score.
Then their 3rd step proceeds to matching. To me the first part of #2 seems flat wrong. Why should your propensity scores be balanced between Treatment and Control before matching?
If you have perfectly balanced propensity scores in the Treatment and Control groups without matching / weighting, doesn't that mean that either:
- You have a randomized experiment or a situation effectively the same as one - congratulations, you do not need to do Propensity Score Matching, or
- You've failed include relevant confounders (observable characteristics which are different between Treatment and Control and which are correlated with the outcome), thus your Propensity Score model failed and your PSM will have no effect in reducing selection bias.
I'd find it much more comforting if the propensity scores were statistically significantly different between Treatment and Control in an observational study where I know that assignment to treatment is non-random, because that would at least imply that the score can be used to control for confounding variation that I might know exists. Am I correct or am I missing something?