I have a button that I can press or not press, a binary target that I would like to be 1 as often as possible, and a bunch of features. I also have a bunch of (feature, button choice, target) data, in which the decision to press the button was made randomly (and thus independently of the other features). I'd like to train something to decide, given a feature vector, whether or not it's a good idea to press the button.
The obvious thing that occurs to me is to make IsButton a feature, train a decision tree/forest. Then, given a feature vector, press the button if and only if the score of the decision tree with IsButton = 1 is greater than the score of the decision tree with IsButton = 0. This produces somewhat sensible results, but it's also very noisy, and I feel like there should be a better way of deciding whether or not to press the button. The binary target is 1 about half of the time, and in aggregate, pressing the button only moves the average value of the target by about 1%. So I'm subtracting two mid-sized numbers to try to estimate a very small difference. IsButton ends up being very low in feature importance, which is fine - IsButton probably isn't the most important factor in terms of influencing the target, I just want to figure out the small effect of pressing the button. But since much of my tree doesn't involve the IsButton feature, I'm spending most of my time building branches, while important for estimating the probability of the target, turn out to have no importance for the actual decision as to whether to press the button. Is there a better training method that reflects the fact that in the end I don't care about the absolute value of the target, but only the effect of pushing the button on the target?