0
$\begingroup$

Suppose I have 3 predictors to predict stock returns. 1 of the 3 is known for ages and is still doing well. The rest 2 are newly found ones. So in a crude portfolio construction fashion, I'd do $s = 0.8s_1 + 0.1s_2 + 0.1s_3$ and then take position proportional to $s$, to reflect that I'm more confident in the first factor than the other two.

Now I have say 50 factors and 5 of them are more traditional and hence more trustworthy (by my subjective opinion). When I try fitting a random forest or xgboost using all the factors, is there a way to tell the model that I trust some factors more than the others?

What I want is to control the resulting prediction to be at least $x\%$ correlated with the traditional factors. And of course, I'm happy to sacrifice some predictive power for that.

$\endgroup$
4
  • $\begingroup$ Hmm... can you maybe reformulate this? Strictly speaking, "at least x% correlated with" is inapplicable for a model that is not explicitly models correlations. In the first instance, why not have two prediction functions, one ($f_{Trad}$) with the 5 "traditional predictors only" and one ($f_{All}$) with all 50 together and then use a voting regressor with weights $[x, 100-x]$? And note that even in that case if for example $f_{Trad}$ outputs $0$, and $f_{All}$ non-zeros, all the variability will be due to $f_{All}$... $\endgroup$
    – usεr11852
    Dec 8, 2023 at 3:22
  • $\begingroup$ Maybe replicate the more trustful columns a couple of times and then use column subsampling in XGBoost (random forest does it by default, except in Scikit-Learn)? $\endgroup$
    – Michael M
    Dec 8, 2023 at 10:00
  • $\begingroup$ @MichaelM: Even then there is no guarantee it will be picked multiple times. Maybe that would work if the OP used Extremely Randomized Forest but that's not on the available algorithms per se. $\endgroup$
    – usεr11852
    Dec 8, 2023 at 10:19
  • $\begingroup$ This is true. However, if we pick features per split, and the column subsampling rate is 1/p (p is the number of features), then it will work better. $\endgroup$
    – Michael M
    Dec 8, 2023 at 10:44

1 Answer 1

1
$\begingroup$

If you want to manipulate model into displaying desired feature importances regardless of the importance, that would have resulted from an honest training, you may try to tinker with Cost Efficient Gradient Boosting. The feature is available in LightGBM and CatBoost (demo), I am not sure about XGBoost. The idea is to specify feature costs, which would be taken into account during split search: a feature would be used for a split only if its gain outweights its cost. Setting high enough costs makes features unlikely to be selected, despite their actual gains.

If you want to make certain features more likely to be selected (without guarantee that they would end up most important), you may

  1. Specify forced splits for certain features. Those would be applied at the root of each tree, before tree fitting begins (forcedsplits_filename in LightGBM)
  2. Fit separate models on the set of all features and subset(s) of traditional features only, then combine them into ensemble
  3. Increase number of histogram bins for selected features (max_bin)

[Added] Monotone constraints is another way to incorporate knowledge about relationships between the target and predictors. iirc XGBoost enforces monotonicity strictly, and LightGBM uses penalization, which is much more flexible.

$\endgroup$
1
  • $\begingroup$ +1 as the CEGB is a good idea but do note that doesn't guarantee that the feature important will be affected in the way we want. For the forced splits I am even more sceptical... if anything they can even be ignored and given that we are loss-guiding our split strategy that means they are even unused. I cannot see how they would ensure a minimal feature importance or even more "prediction correlation". The model ensembling suggestion suffers from what I mentioned in my original comment. But nice additional suggestion on CEGB! (Not a full solution but cool) $\endgroup$
    – usεr11852
    Dec 8, 2023 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.