I am trying to measure the effect of a treatment on a binary outcome using observational data. However, the group that was treated and the group that was not treated are not equivalent: assignment to the treatment or control group is not randomized and probably depends on many different variables.

I want to perform regression adjustment to estimate the effect of treatment while taking into account confounding variables, in order to have an idea of the "true" effect of my treatment. Essentially, I want to fit a logistic regression model where the dichotomous outcome is explained by treatment and other confounding variables. Then I want to look at the coefficient of treatment to estimate treatment effect.

My main question is: how can I know whether my model is satisfactory? I don't know what the "true" confounding variables are and I have access to a large volume of data and predictors. However, I don't know when I can safely say that my model has efficiently corrected for the structural bias of the data. I'm thinking about reading the pseudo R squared in the logistic regression results, but I'm not sure what a "good" value would be. I'm also wondering if there are other methods to assess that my model correctly estimates treatment effect.


In general, there may be any number of confounders and they may be known or unknown. Nothing inside your data will tell you, whether some key confounders are not known to you and this instead requires subject matter knowledge. There are a number of methods for adjusting for potential confounders including covariate adjustment (which you seem to consider) and propensity score methods (e.g. matching by observations by propensity or stratification by propensity score - using them as covariates in the model is rather problematic).

Assuming you think you have information on all the most important potential confounders, you may face the problem that you may not have enough data for it to be clear, which ones of these may play a role and in what form (for continuous ones, you may of course have any kind of non-linear effect). p>0.05 or 0.15 or AIC1 > AIC2 or pseudo R squared based on a limited sample size is in no way a criterion that could be appropriate for justifying that we do not need to adjust for a potential confounder. In addition, model building causes severe issues with inference (if you go for covariate adjustment) and you would have to do a lot of complicated adjustments (e.g. bootstrapping the whole model building process) to make sure the inference you obtain for the effect of interest is in anyway meaningful. Apparently, propensity score matching is a lot more robust to overfitting the propensity score model, you could use all kinds of spline methods and may for that reason be of interest. It also has the nice feature that you can check whether the covariates across matched pairs or strata are reasonably balanced.

Observational data analysis is one of the more challenging statistical tasks and there are a number of good books (e.g. the one by Rosenbaum) and key articles (e.g. by Rubin and/or Rosenbaum) that I would recommend.

  • $\begingroup$ Thank you Bjorn. I have read a lot about propensity score matching and stratification. I have tried the latter by running a regression with treatment as the outcome and some confounding variables as predictors. I did not find very satisfactory results with this method either, however. Most of my variables are dichotomous and I think that had a pretty negative effect on my stratas (propensity stores quintiles), which in the end did not seem sufficiently balanced. Could this be because my propensity score prediction was not sufficiently robust? $\endgroup$ – Konrad May 16 '17 at 7:57
  • $\begingroup$ I'll also be taking a look at your references. By the way, I have very large volumes of data, if that helps, though I'm trying to measure pretty small treatment effects... $\endgroup$ – Konrad May 16 '17 at 7:57
  • $\begingroup$ I am unaware of the way "spline methods" work. I've read a lot about the term on StackExchange but I'm not sure how to implement that in a propensity score stratification or matching method. $\endgroup$ – Konrad May 16 '17 at 7:59

You might find the below reference of use:


Essentially, a good first step to estimate the impact of potential confounders on both your dependent and independent variable of interest, is to add confounding variables to your regression in a stepwise fashion.

For instance, if you find that upon adding a confounder that your coefficient changes value by a significant percentage, e.g. 10% or more, then you can infer that the added variable is indeed a confounder.

Moreover, take into account the effect of interaction when looking at the differences between your treatment and control groups. e.g. if sex is one of your variables and your interaction variable is (treatment*sex), then you could find that such a variable is statistically significant, indicating that treatment differs by gender.

For a logistic regression, you are likely better off using the AIC value to compare models. The lower the AIC value, the better the indicated fit.

Hope this helps.

  • $\begingroup$ While this is interesting, I'm not sure that the AIC value is a good to way to know whether my model properly corrects for structural bias in my data. A quick look on Wikipedia said that the AIC value is good for comparing models but that it does not give any absolute knowledge on model quality. $\endgroup$ – Konrad May 16 '17 at 7:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.