A theoretical question. I am tasked to estimate the effectiveness of treatment A in comparison to treatment B. Let's say these treatments are in the field of risk management so their targets are anything but randomly chosen. However, the reasons for choosing treatment A over treatment B are opaque and differ from case to case so there are no obvious set of covariates. We do have a hunch of which attributes have a say in treatment selection so we have created a dataset containing ~ 40 covariates. Half-decent model can be built using these to predict whether observation X gets treatment A or B so propensity scores are a good and possible option and we are clearly not shooting in the wild.
I don't have a lot of data so I prefer to use methods which don't lose data (so no CEM matching for example) so have used different weighting schemes for weighted regression and different propensity score matching-set ups, also tried matching on principal components. Double ML and causal forests produced really bad results and basically were unable to differ from the raw, unbalanced difference in results between treatments so those I no longer consider.
Here comes my question: can I just opportunistically try a whole bunch of different covariate balancing schemes and just pick the one which seems to produce the best results, eg most balanced data set? Or should I first try to narrow down the set of covariates on which the balance is desired? (balancing a covariate which is in no way related to getting treatment is moot). When different schemes might produce even quite wildly different estimates, how to gauge which is the most plausible one?
I do know that treatment A produces better results than treatment B, both data and theory support it. Now just to find the most plausible estimate on exactly how much and to be able to defend my choices :p