What is the link between methods such as matching and statistically controlling for variables?

Question

Often in research articles you read the researchers have controlled for certain variables. This can be done by methods such as matching, blocking, etc.

But I always thought controlling for variables was something done statistically by measuring several variables that could be of influence and performing some statistical analysis on those, which could be done in both true and quasi experiments. So, for instance you would have a survey or other test in which you´d measure the independent variable and some possibly confounding variables and do some analysis.

• Is is possible to control for variables in quasi experiments?
• What is the link between methods such as matching and statistically controlling for variables?

Answer 1

As with AdamO, I think the key to answering this question is the notion of causal inference, and how to get "toward" a causal model using observational setups.

In a perfect world, we would have something called a counterfactual population - the study population, identical in all respects except for the single thing we are interested in. The difference between those two populations, based on that difference, is a true causal result.

Obviously, we can't have this.

There are ways however, to try to get close to it:

• Randomization: This theoretically (if randomization is done correctly) should give you two populations that are identical, except for treatment post-randomization.

• Stratification: You can look at a population within levels of covariates, where you are making "like with like" comparisons. This works splendidly for small numbers of levels, but quickly becomes cumbersome.

• Matching: Matching is an attempt to assemble a study population such that Group A resembles Group B, and thus is amenable to comparison.

• Statistical adjustment: Including covariates in a regression model allows for the estimation of an effect within levels of the covariates - again, comparing like with like, or at least attempting to.

All are an attempt to get closer to that counterfactual population. How to best get at it depends on what you want to get out, and what your study looks like.

Answer 2

I think causal modeling is the key to answering this question. One is faced at the outset to identify the correct adjusted/stratified/controlled effect of interest before even looking at data. If I were to estimate the height / lung capacity relationship in adults, I would adjust for smoking status since smoking stunts growth and influences lung capacity. Confounders are variables which are causally related to the predictor of interest and are associated with the outcome of interest. See Causality from Judea Pearl, 2nd ed. One should specify and power their analysis for the correct confounding variables before the data collection process even begins using rational logic and prior knowledge from previous exploratory studies.

This doesn't mean, however, that some researchers don't rely on data-driven methods to select adjustment variables. I don't agree with doing this in practice when conducting confirmatory analyses. Some common techniques in model selection for multiple adjusted models is forward/backward model selection where you can restrict to classes of models which you believe to be at least plausible. The blackbox AIC selection criteria for this is related to the likelihood and, hence, the degree of reduction in the $R^2$ for linear models for these adjustment variables. Another process common in epidemiology is where variables are only added to the model if they change the estimate of the main effect (like an odds ratio or hazard ratio) by at least 10%. While this is "more" correct than AIC based model selection, I still think there are major caveats in this approach.

My recommendation is prespecify the desired analysis as part of a hypothesis. The age adjusted smoking / cancer risk is a different parameter, and leads to different inference in a controlled study than the crude smoking / cancer risk. Using subject matter knowledge is the best way to select predictors for adjustment in regression analyses, or as stratification, matching, or weighting variables in various other types of "controlled" analyses of experimental and quasiexperimental design.

Answer 3

The story about the relationship between matching and regression is briefly summarised in a blog post here. In short

"Regress on D [a treatment indicator] an a full set of dummies (i.e., saturated) model for X [covariates]. The resulting estimate of the effect of D is equal to matching on X, and weighting across covariate cells by the variance of treatment conditional on X"

See also section 3.3 of Mostly Harmless Econometrics or section 5.3 of Counterfactuals and Causal Inference for a thorough discussion, including the pros and cons of the D given X weighting that regression implicitly provides.

@EpiGrad gives a good start on your first question. The books linked above treat it almost exclusively. If you do not have a computer science / math background you may find Pearl hard going (although worth it in the end!)