When is there enough or too much balancing in observational causal studies? I'm trying to compare exam performance across genders when I match on a variety of students' characteristics (e.g., their age, parental income etc). I have many such matchable variables.
My question: is there is a possibility that if I use "too many" matching variables I may leave so little remaining variability in the data that it's impossible or difficult to discern any true differences between the genders?
 A: Short answer: yes.
Longer answer: matching or blocking on a variable usually winds up in the analysis as that variable showing up in the right-hand side of the linear model or ANOVA. This is tantamount to "conditioning on" that variable. When should you condition, and when should you not condition? As you have already tagged this question with the "causality" tag, it will hopefully not surprise you to know that the tools of the New Causal Revolution, particularly those of the DAG approach of Judea Pearl, can help answer the question of when to condition, and when not to condition. At a minimum, you should condition on confounders, and you should NOT condition on mediators. A confounder is a variable that sets up a backdoor path, like this:

$Z$ is a confounder for the causal effect of $X$ on $Y.$ You must condition on (block on) $Z$ to get the correct causal effect of $X$ on $Y.$
A mediator is similar in some ways, but requires a radically different approach. A mediator happens when some of the causal influence goes through (is mediated by) another variable, like this:

$Z$ here is a mediator for the causal effect of $X$ on $Y.$ If you condition on $Z$ in this scenario, you will get the incorrect (biased) effect of $X$ on $Y.$ In addition, you might very well end up choking off all the causal effect, if it's all mediated through $Z.$
In summary: it pays to come up with a careful causal diagram, and to reason from that diagram. It can help design better experiments, and it can help you analyze observational studies better.
