I have a basic question as I am new to Data Science. I have a typical problem where I need to estimate the effect of an intervention between experimental and control groups. I read propensity matching is the way to go to compare 'like' candidates and people use Logistic regression (LR) to calculate propensity scores. My basic question is, isn't LR a supervised machine learning technique? What will be my training data for LR? In other words, how can LR predict the probability of being assigned to either groups just based on some properties about the candidates?

  • 2
    $\begingroup$ It is not appropriate to engage in inference about interventions without semi-intensive study of statistics. There are many, many fundamentals you need to know before proceeding including when to use propensity analysis vs. straight covariate adjustement, confounding, outcome heterogeneity, non-collapsibility of odds ratios, problems with throwing away data with matching algorithms, why prop. analysis is a data reduction method, why prop. does not use the outcome variable, .... For a rough start see the observational treatment comparison chapter in BBR:fharrell.com/p/blog-page.html $\endgroup$ – Frank Harrell Aug 13 '17 at 13:38
  • $\begingroup$ Thanks, Frank. While I am going through the documentation, it will be helpful if you could explain what exactly am I missing when I wonder- how can LR calculate propensities of candidates in a population to be assigned to either of the groups with no training data? Or is it really that it is just grouping like candidates together regardless of what the outcomes mean? $\endgroup$ – Prav Aug 13 '17 at 14:01
  • 1
    $\begingroup$ You have to study and read. As a statistician if I wanted to understand nephrology I would start working with a nephrologist. Start by forgetting about the idea of training and test data. $\endgroup$ – Frank Harrell Aug 13 '17 at 14:03
  • $\begingroup$ I would but everywhere I read logistic regression it does talk about training data and it's use to minimize the cost function. As for working with a statistician, that's why I am here :) $\endgroup$ – Prav Aug 13 '17 at 14:49
  • 1
    $\begingroup$ You need a local statistician and you are confusing model validation with inference. The use of the word documentation instead of textbook suggests you want to shortcut the process. $\endgroup$ – Frank Harrell Aug 13 '17 at 15:32

The test data is the training data. Logistic regression uses the observed relationships between the predictors the observed treatment assignment to estimate numerical relationships between those predictors and the probability (actually the odds) of treatment assignment. Then once you have those numerical relationships, you apply them to the same data set used to estimate the relationships to generate the predicted probabilities.

Overfitting is not really a problem here because the goal or propensity score analysis is not good prediction but rather covariate balance (where each covariate has the same distribution across treatment groups). There actually are machine learning methods to estimate propensity scores: generalized boosted modeling is the most popular. Again, because the test data is the training data, you must impose a stopping condition other than good prediction to arrive at valid propensity scores; that stopping condition is usually a numerical summary of covariate balance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.