I'm working with a dataset of ~100,000 individuals where ~500 (0.5%) individuals received treatment.

I have several continuous and count outcomes for all observations that I would like to compare between treated and untreated. It would be important for the analysis to match individuals on several characteristics (that could be binary, continuous or categorical).

I'm working with Stata.

I was brainstorming several possible scenarios that include:

  1. stratified analyses of treated and untreated

  2. treating it as case control study and attempting to match 'controls' to all my 'cases' where criteria of match allow. Then moving forward with analysis appropriate for that set up (conditional logit would work for binary outcomes.. not sure about continuous and count ones..)

  3. Treatment-effects estimation, perhaps using propensity-score matching (not sure if and how it is possible to include categorical variables though..)

What analysis would be most appropriate for such dataset?

  • 1
    $\begingroup$ It would be helpful if you could elaborate what several means? 5, 10, 30? Do you know much overlap and balancing is in your data between treatment and control group? I think the Gelman and Hill is a good textbook to start with. For matching, see stat.columbia.edu/~gelman/arm/chap10.pdf $\endgroup$ – Arne Jonas Warnke Dec 2 '16 at 9:45
  • $\begingroup$ @ArneJonasWarnke Minmum (from epidemiological point of view) would be - sex, age (continuous or categorized) & ~6-10 main disease types. $\endgroup$ – radek Dec 2 '16 at 10:20
  • 1
    $\begingroup$ Since your control group is so large, can you do perfect matching on (at least some of) those characteristics / pre-existing diseases? This eliminates all biases due to differences in those characteristics. You can also combine perfect matching (in a first stage) with propsensity score matching, see for example faculty.unlv.edu/nasser/ECO%20772,%20Econometrics%20II/… $\endgroup$ – Arne Jonas Warnke Dec 2 '16 at 10:31
  • $\begingroup$ Thanks @ArneJonasWarnke. Assuming such matched dataset (and discarding unmatched controls) - what would be the best method for such scenario? $\endgroup$ – radek Dec 2 '16 at 12:02

Removing good data from an analysis is scientifically suspect in my humble opinion, and naive matching methods are inefficient. It may be very easy to adjust for patient characteristics using ordinary regression models, paying attention to linearity assumptions etc. Of course it is a good idea to look at overlap in covariate distributions across treatment groups to see where assumptions of no interaction between treatment and characteristics might be on shaky ground and untestable.

| cite | improve this answer | |
  • $\begingroup$ Thanks @Frank. Should I then simply start considering models using all data and use an estimate on a binary categorical variable (received treatment) as an estimate of difference for treated group? $\endgroup$ – radek Dec 6 '16 at 9:07
  • 1
    $\begingroup$ Yes with careful and fairly liberal covariate adjustment. Propensity score analysis is for the opposite situation where the outcome is what is rare. $\endgroup$ – Frank Harrell Dec 6 '16 at 11:18
  • $\begingroup$ And how about situations when certain combinations of sex, age and disease type when no treatment was found? Should they be excluded beforehand? Or kept in the model (since they will not provide any information I believe)? Also the opposite situation might raise a challenge - there are few rare cases when age/sex/disease combination has disease only, but no observations without disease.. $\endgroup$ – radek Dec 6 '16 at 13:39
  • 1
    $\begingroup$ You have to decide which variables are unlikely to interact with the other variables. When effects are additive you don't need to have all combinations of values well represented in the data. $\endgroup$ – Frank Harrell Dec 6 '16 at 21:10
  • $\begingroup$ Missed the deadline for the bounty. I haven't managed to solve the problem but @Frank's solution will most likely be the preferred one. Thanks for help! $\endgroup$ – radek Dec 11 '16 at 16:06

Based on the comments and the availability of such a large control group, I would probably advise to do in a step first exact matching on age groups and sex, and perhaps common disease groups. Hereby, you built different strata. In a second step, you can apply propensity score matching to ensure that treatment and control group are as balanced as possible with respect to the remaining observables.

You can do this apparently using the psmatch2 package for Stata (I have used that package only briefly out of interested).

A code example is given in the help file:

    g att = .
    egen g = group(groupvars)
    levels g, local(gr)
    qui foreach j of local gr {
            psmatch2 treatvar varlist if g==`j', out(outvar)
            replace att = r(att) if  g==`j'

    sum att

See here for further information


You should -- of course -- verify that there is enough overlap between treatment and control group within each strata.

* Update: Response to the comment of Frank Harrell *

Why I argue for matching:

It is a trade-off between, on the one hand, balancing covariates as close as possible between treatment and control group, and, on the other hand, removing data (what Frank Harrell emphasized).

It is clear that the estimator becomes in a first step less efficient if you remove data, and you should justify ignoring data. But radek has asked for matching approaches and I agree that this is a good idea.

Matching avoids to some extent "extrapolation bias" if the covariate distribution differs between treatment and control group. You drop observations which give you few or any information about the treatment effect because their covariates are very far away from the sample.

Many prominent researchers therefore recommend matching or subclassificatioon plus regression.

See Imbens & Rubin, Ho et al., Imbens & Wooldridge

| cite | improve this answer | |
  • $\begingroup$ I don't see why matching plays a role here. $\endgroup$ – Frank Harrell Dec 11 '16 at 16:11
  • $\begingroup$ I find this argument not convincing. Matching on continuous variables results in an incomplete adjustment because the variables have to be binned. Matching throws away good data from observations that would be good matches. Extrapolation bias is only a significant problem if there is a covariate by group interaction, and users of matching methods ignore interactions anyway. If you don't want to make regression assumptions that are unverifiable, remove observations outside the overlap region just as with matching. $\endgroup$ – Frank Harrell Dec 12 '16 at 12:07

The propensity score (PS) is a balancing score indicating the probability of treatment assignment conditional on observed baseline characteristics. In a randomized controlled Trial (RCT) the PS is known. Estimation and application of PS therefore, mimic some of the particular characteristics of an RCT and therefore are method of choice to estimate the treatment effect in an observational study, provided that you have no unmeasured confounders and all subjects have a non-zero probability of receiving Treatment.

You can estimate the PS using logistic regression or generalized boosting methods and therefore, include all observed (continous and categorical) baseline variables. Notably, you do not include the outcome parameter. (There is an ongoing debate, which variables to include, however, Austin postulates it is safe to include all observed baseline variables https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3144483/). It is important to underscore that you do not aim to find the best "predictive model" but instead the model that balances your covariates best. After application of the PS using stratification, matching, inverse probability Treatment weights or covariate adjustment (all of them have pros and cons and are somewhat dependent on your data), you can test the treatment effect on your outcomes.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.