0
$\begingroup$

I have a button that I can press or not press, a binary target that I would like to be 1 as often as possible, and a bunch of features. I also have a bunch of (feature, button choice, target) data, in which the decision to press the button was made randomly (and thus independently of the other features). I'd like to train something to decide, given a feature vector, whether or not it's a good idea to press the button.

The obvious thing that occurs to me is to make IsButton a feature, train a decision tree/forest. Then, given a feature vector, press the button if and only if the score of the decision tree with IsButton = 1 is greater than the score of the decision tree with IsButton = 0. This produces somewhat sensible results, but it's also very noisy, and I feel like there should be a better way of deciding whether or not to press the button. The binary target is 1 about half of the time, and in aggregate, pressing the button only moves the average value of the target by about 1%. So I'm subtracting two mid-sized numbers to try to estimate a very small difference. IsButton ends up being very low in feature importance, which is fine - IsButton probably isn't the most important factor in terms of influencing the target, I just want to figure out the small effect of pressing the button. But since much of my tree doesn't involve the IsButton feature, I'm spending most of my time building branches, while important for estimating the probability of the target, turn out to have no importance for the actual decision as to whether to press the button. Is there a better training method that reflects the fact that in the end I don't care about the absolute value of the target, but only the effect of pushing the button on the target?

$\endgroup$
  • 3
    $\begingroup$ Sorry, I'm having a little bit of trouble following the initial scenario you describe. As well as everything else. It's also not quite clear to me why you would need to build two trees and not just one... Could you please describe--in concrete terms--what your data actually looks like? $\endgroup$ – Steve S Aug 12 '14 at 1:19
1
$\begingroup$

I think you might want to try a logistic regression model. basically you are after an accurate estimate of the probability of target ( rather than simple classification).
That is not what you are going to get from a standard tree model or SVM.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

You might counteract the noise by learning very high number of trees. If pressing the button only moves the average value of the target by about 1%, then maybe you need around $10^4$ trees.

If possible, you should obtain a bunch of triplets (x, Target(Features=x, IsButton=1), Target(Features=x, IsButton=0) ), i.e. comparing the result when the button is on and when the button is off for the same set of features. Then you learn the function Targer(x,1)-Target(x,0).

Another way of action is to shift from the decision forest to another learning model, i.e. neural network, which is smoother and can be directly penalized for dependence on IsButton.

The last idea is to learn IsButton as a function of Features and Target, instead of learning Target as a function of Features and IsButton.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.