How are propensity scores different from adding covariates in a regression, and when are they preferred to the latter? I admit I'm relatively new to propensity scores and causal analysis.
One thing that's not obvious to me as a newcomer is how the "balancing" using propensity scores is mathematically different from what happens when we add covariates in a regression? What's different about the operation, and why is it (or is it) better than adding subpopulation covariates in a regression?
I've seen some studies that do an empirical comparison of the methods, but I haven't seen a good discussion relating the mathematical properties of the two methods and why PSM lends itself to causal interpretations while including regression covariates does not. There also seems to be a lot of confusion and controversy in this field, which makes things even more difficult to pick up.
Any thoughts on this or any pointers to good resources/papers to better understand the distinction? (I'm slowly making my way through Judea Pearl's causality book, so no need to point me to that)
 A: A likely obtuse reference, but if you by chance have access to it I would recommend reading this book chapter (Apel and Sweeten, 2010). It is aimed at social scientists and so perhaps not as mathematically rigorous as you seem to want, but it should go into enough depth to be more than a satisfactory answer to your question.
There are a few different ways people treat propensity scores that can result in different conclusions from simply including covariates in a regression model. When one matches scores one does not necessarily have common support for all observations (i.e. one has some observations that appear to never have the chance to be in the treatment group, and some that are always in the treatment group). Also one can weight observations in various ways that can result in different conclusions.
In addition to the answers here I would also suggest you check out the answers to the question chl cited. There is more substance behind propensity scores than simply a statistical trick to achieve covariate balance. It you read and understand the highly cited articles by Rosenbaum and Rubin it will be more clear why the approach is different than simply adding in covariates in a regression model. I think a more satisfactory answer to your question is not necessarily in the math behind propensity scores but in their logic.
A: One big difference is that regression "controls for" those characteristics in a linear fashion. Matching by propensity scores eliminates the linearity assumption, but, as some observations may not be matched, you may not be able to say anything about certain groups.
For example, if you are studying a worker training program, you may have all the enrollees be men, but the control, non-participant population be composed of men and women. Using regression, you could regress, income, say, on a participation indicator variable and a male indicator. You would use all your data and could estimate the income of a female had she participated in the program.
If you were doing matching, you could only match men to men. As a result, you wouldn't be using any women in your analysis and your results wouldn't pertain to them.
Regression can extrapolate using the linearity assumption, but matching cannot. All the other assumptions are essentially the same between regression and matching. The benefit of matching over regression is that it is non-parametric (except you do have to assume that you have the right propensity score, if that is how you are doing your matching).
For more discussion, see my page here for a course that was heavily focused on matching methods. See especially Causal Effects Estimation Strategy Assumptions.
Also, be sure to check out the Rosenbaum and Rubin (1983) article that outlines propensity score matching.
Lastly, matching has come a long way since 1983. Check out Jas Sekhon's webpage to learn about his genetic matching algorithm.
A: The short answer is that propensity scores are not any better than the equivalent ANCOVA model, particularly with regard to causal interpretation.
Propensity scores are best understood as a data reduction method.  They are an effective means to reduce many covariates into a single score that can be used to adjust an effect of interest for a set of variables.  In doing so, you save degrees of freedom by adjusting for a single propensity score rather than multiple covariates.  This presents a statistical advantage, certainly, but nothing more.

One question which may arise when using regression adjustment with
  propensity scores is whether there is any gain in using the propensity
  score rather than performing a regression adjustment with all of the
  covariates used to estimate the propensity score included in the
  model.  Rosenbaum and Rubin showed that the "point estimate of the
  treatment effect from an analysis of covariance adjustment for
  multivariate X is equal to the estimate obtained from a univariate
  covariance adjustment for the sample linear discriminant based on X,
  whenever the same sample covariance matrix is used for both the
  covariance adjustment and the discriminant analysis". Thus, the
  results from both methods should lead to the same conclusions.
  However, one advantage to performing the two-step procedure is that
  one can fit a very complicated propensity score model with interactions
  and higher order terms first. Since the goal of this propensity score
  model is to obtain the best estimated probability of treatment
  assignment, one is not concerned with over-parameterizing this model.

From:
PROPENSITY SCORE METHODS FOR BIAS REDUCTION IN THE COMPARISON OF A TREATMENT TO A NON-RANDOMIZED CONTROL GROUP
D'Agostino (quoting Rosenbaum and Rubin)
D’agostino, R.B. 1998. Propensity score matching for bias reduction in the comparison of a treatment to a non-randomized control group. Statistical Medicine 17: 2265–2281.
A: I like to think of PS as a design portion of the study which completely separated from the analysis. That is, you might want to think in terms of design (PS) and analysis (regression etc...). Also, PS porvides a mean of supporting exchangeability for binary treatment; maybe others can comment on whether including the covariates in the outcome model can actaully support exchangebility, or whether one assume exchangeability prior to including the covariates in the outcome model.
A: Stat Methods Med Res. 2016 Apr 19. 
An evaluation of bias in propensity score-adjusted non-linear regression models.
Propensity score methods are commonly used to adjust for observed confounding when estimating the conditional treatment effect in observational studies. One popular method, covariate adjustment of the propensity score in a regression model, has been empirically shown to be biased in non-linear models. However, no compelling underlying theoretical reason has been presented. We propose a new framework to investigate bias and consistency of propensity score-adjusted treatment effects in non-linear models that uses a simple geometric approach to forge a link between the consistency of the propensity score estimator and the collapsibility of non-linear models. Under this framework, we demonstrate that adjustment of the propensity score in an outcome model results in the decomposition of observed covariates into the propensity score and a remainder term. Omission of this remainder term from a non-collapsible regression model leads to biased estimates of the conditional odds ratio and conditional hazard ratio, but not for the conditional rate ratio. We further show, via simulation studies, that the bias in these propensity score-adjusted estimators increases with larger treatment effect size, larger covariate effects, and increasing dissimilarity between the coefficients of the covariates in the treatment model versus the outcome model.
A: Like the person who asked the question, I am relatively new to propensity score analysis.  However, my scientific collaborator has deep knowledge and expertise in biostatistics and clinical trial analysis, so I posed this question to him.  His answer provides additional insight beyond what was already posted:
When using observational data to estimate the causal effect of a primary exposure (e.g. a drug treatment or intervention) on a health outcome, it is important to balance the treatment and control groups for any other exposures.  This balancing attempts to mimic the effect of randomization, which is not possible with observational data.  Some exposures are associated with both the treatment and the outcome and thus, meet the definition of confounders.  Confounders can be adjusted in multivariable regression models of the health outcome.  However, other exposures are associated with the treatment, but NOT the health outcome, and cannot be adjusted in regression models of the outcome.  Systematic differences between the treatment and control groups can lead to treatment selection bias.
The proper way to balance the exposures between the treatment and control groups is to use propensity score matching, without consideration of the health outcome variable. Propensity score matching balances the exposures between the treatment and control groups above and beyond what can be accomplished by multivariable regression modeling of the health outcome.
Here is a helpful tutorial, with references, on propensity score analysis from Columbia University's Mailman School of Public Health:  https://www.publichealth.columbia.edu/research/population-health-methods/propensity-score-analysis.
Best wishes,
Dave
