1
$\begingroup$

I have a question related to the estimation of effects of a certain binary exposure variable on a binary outcome via logistic regression after an exact matching procedure.

Originally, I followed recommendations in the literature and in the MatchIt library documentation to include as controls the variables I used for matching, in order to obtain more precise estimates of my X of interest. I derived odds ratios for a series of specification models, and that was fine (and this was the ATT).

Yet, I also wanted to derive the marginal effect for my X or, more generally, the difference in the predicted probability of having 1 as outcome across the two levels of my X (X=0 not treated, X=1 treated). When I use the margins library (as well as the ggpredict function), after fitting the logistic regression model with control variables, I obtain a very narrow difference in probability, and in general a tiny marginal effect. When using ggpredict, I see that all control variables are adjusted by setting them as fixed on some categories (please consider that most of my matching/control variables are categorical). Yet, I feel this is not what I want, I do not care about the marginal effect computed when fixing values to particular categories.

So I tried to avoid controls in regression: I only estimated the effect of X on Y using weights after matching. When I do this, the odds ratios are fairly the same as the ones in the first model with controls, but the difference in predicted probabilities as well as the marginal effects increase. What do you think is the best approach?

I know this might depend on several aspects I haven't detailed here, but the main point here is that while ORs do not change across the two main specifications (with and without controls), the marginal effects seem to change. In the case with controls, the predicted probabilities as well as marginal effects are too small to make sense theoretically. Not only odds ratios were pretty large and significant (although I am aware of the problems of interpreting odds ratios in terms of magnitude across different models), but also the difference in means for the treated and non treated groups are quite considerable, so why this tiny marginal effects?

Any advice on how to proceed? Please again consider that I used exact matching on 10/12 variables, with 8/10 of these being categorical, only 2 being continuous. Hence, I obtained perfect balance across the two groups. Thank you for any input/suggestion!


Here are some details on the models, as requested by Noah. The variables are anonymized, X is the treatment variable, and coefficients minimally changed (due to privacy restrictions on the project I am working on).

In the case with controls, after exact matching, this is the code I used:

model_matched2 <- glm(Y ~ X + 
                    A + B + C + D + E + F, 
                  family=quasibinomial(link=logit), data = matched_dataset2, 
                  weights=weights)

And these are the results -obtained through the "summ" function, with robust standard errors - for the odds ratios (I only report the X one):

X OR: 0.815
95% CI: [0.806-0.825]
z value: -34.091
p- value: 0.000

Then I calculate the predicted probabilities, via

ggpredict(model_matched2, variables="X")

And I get 0.50 for non-treated, 0.53 for treated, of course adjusted for the various controls in the estimated equation.

When I estimate the marginal effect, via

margins(model_matched2u, variables="X")

the estimated marginal effect is 2.11.

When avoiding controls, I use the following code:

model_matched2u <- glm(Y ~ X, 
                  family=quasibinomial(link=logit), data = matched_dataset2, 
                  weights=weights)

At this point, the results in OR form, again with robust st. errors, are:

OR: 0.871
95% CI: [0.862-0.879] z value: -28.060
p-value: 0.000

Marginal effect is -0.07749, estimated via:

margins(model_matched2u)

Predicted probabilities are 0.66 for treated, 0.74 for non-treated, estimated via:

 ggpredict(model_matched2u)

This is an example, I have different models, different specifications, but this kind of pattern (similarity in OR, difference in margins, with models with controls having seemingly unreasonable small marginal effects) keeps popping up.

Any hint would be fantastic! Thanks

$\endgroup$
6
  • $\begingroup$ Matching is not hard to do but the analysis needing to be chosen for the matched data is complex and not always very well thought out in the literature. Since matching results in deleting good data and causes great statistical inefficiency (power loss) you have to work hard to justify the use of matching in this context. Matching also destroys any marginal effects you want to estimate because matching is not faithful to the original sampling scheme. $\endgroup$ Sep 23 at 12:21
  • $\begingroup$ Hi Frank. Matching here is somehow justified by the large dataset I am working with, so yes while I discard some thousands of observations in absolute terms, these account for only 5-20% of all observations. Hence, power loss becomes less of a problem. Plus, I would not talk about "sampling scheme" because I am virtually using data for the entire population of observations I am interested in. At any rate, if you have suggestions about how to proceed regarding the statistical analysis, I'd be happy to hear your thoughts! $\endgroup$ Sep 23 at 13:19
  • $\begingroup$ Would you be willing to post some of your models and results? It would be really helpful to see the logistic regression models, the coefficients on treatment in those models, and the marginal effect specifications, estimates, and SEs. $\endgroup$
    – Noah
    Sep 23 at 14:37
  • $\begingroup$ Sure @Noah, let me add them as an answer to my question. $\endgroup$ Sep 23 at 14:51
  • $\begingroup$ I'm still not getting why matching is a good idea in this setting especially if you discard > 15% of observations. Large datasets justiify direct adjustment through regression, which adjust through explanation of outcome variation, increasing power. $\endgroup$ Sep 24 at 11:25

1 Answer 1

1
$\begingroup$

This is a problem with the margins package. You should use the marginaleffects package instead, which makes the analysis simple and uses the correct defaults for this problem.

To get a marginal risk difference, whether you have covariate adjustment or not, you will run the following code:

comparisons(fit, variables = "X", newdata = subset(matched_dataset2, X == 1)) |>
  summary()

Take a look at that output. A clue that there is a problem in the margins package is that you got a risk difference of 2.11, which obviously makes no sense; it was performing the marginal effects on the linear scale, which is not what you want.

If you want a marginal log odds ratio using marginaleffects, you can do the following:

comparisons(fit, variables = "X", newdata = subset(matched_dataset2, X == 1),
            transform_pre = "lnoravg") |>
  summary()

For the unadjusted model, this should return the original regression coefficient. Not that you might need to be using cluster-robust standard errors.

(For future readers, if you are estimating the ATE, remove the newdata argument and add wts = "weights".)

$\endgroup$
6
  • $\begingroup$ Thanks for the thorough comment, Noah. I have tried to follow your suggestions, but the results do not seem to change. The OR remain the same, the predicted probabilities using ggpredict as well. When I calculate marginal effects via margins as well as marginaleffects, the effect is small. It seems unreasonable to think that passing from non-treated to treated only reduces the probability of having 1 as outcome by 1-2-3%, especially because the difference in means for the outcome and the groups are substantial. Any additional hint? Thanks $\endgroup$ Sep 24 at 10:30
  • $\begingroup$ Also, for some dataset, I get the '"Error in rbindlist(lo, fill = TRUE) : Column 21 of item 1 is length 11 inconsistent with column 1 which is length 41756. Only length-1 columns are recycled." after comparisons. Any clue? $\endgroup$ Sep 24 at 10:37
  • $\begingroup$ What is the intercept in the unadjusted logistic regression in the matched sample? $\endgroup$
    – Noah
    Sep 24 at 14:27
  • $\begingroup$ It's 2.217 (p>0.00001, tval=226.422) $\endgroup$ Sep 24 at 14:45
  • $\begingroup$ Okay! This is critical info. An intercept of 2.217 means the risk under control is 0.902, and an odds ratio of 0.871 means the risk under treatment is .88. The marginal effects analysis lines up perfectly with the unadjusted odds ratio analysis. So I don't see what the discrepancy is. ggpredict has nothing to do with estimating treatment effects, so there is no reason to use it, and no part of my answer was meant to change the OR results, just explain to you how to estimate marginal effects using a different package. I see no discrepancy. $\endgroup$
    – Noah
    Sep 24 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.