Using Random Forest vote scores as a matching variable?

Question

I am working on a project where we need to identify good counterfactuals / matches for a binary treatment, which is regressed against a binary outcome. The "treatment" that we seek to study is quite fuzzy, and there is no certain way to decide on specific matching variables without inviting to significant critique.

In this, I am thinking about using a Random Forest algorithm to produce voting scores for each individual and year in the dataset, based on a barrage of information available on them for each year, and then select the counterfactual to each treated individual based on other individuals that have an equal voting score for the end outcome (interpreted as probability of yielding the same end-outcome).

In this process, I now have the following two question:

1. Does it make sense to use voting scores as a basis for selecting counterfactuals? (i.e., in similarity to using a conventional PSM or CEM)

2. Are there any literatures that contain examples of this? I have searched Google Scholar far and wide, but the only things that I have found are on image processing.

Edit: All vote scores are also transformed using Platt scaling

Sincerely Johan

Answer 1

I am no expert on random forests, so I can't comment on their specific use, but it is definitely possible to perform matching on the predicted values of a model trained by regressing the outcome on the covariates. The predicted values are called prognostic scores, and this method is called prognostic score matching. I go into some detail on the advantages and disadvantages of using this method in this post.

To summarize, prognostic scores often perform better than propensity scores as matching variables, and prognostic and propensity scores can be used together highly effectively. Prognostic score matching breaks the separation between the design and analysis of an observational study and does so in a potentially suboptimal way; if you're going to fit a model to the outcome, it can be better to just use that model to generate counterfactuals and use them to estimate the effect rather than to use them in matching. You will have to bootstrap the entire process of model fitting, matching, and effect estimation to get remotely valid standard errors.

If you want to read more about this, I provide a few citations in my linked answer. Some citations I missed in that answer are Leacy and Stuart (2014), who discuss the joint use of propensity and prognostic scores for matching and which I discuss briefly here, and Waernbaum (2012), who compares the robustness properties of several methods including prognostic score matching.