Consider a simplified example where we are trying to predict some value
val which happens to be equal to one of the
num_* features. The task of the model is to select which of these
num_* features to focus on, depending on the values of other features.
What the model would learn in this toy example, is that when we have a
blue, large, rectangle object, then
val would mostly be equal to
num_2, while when having a
red, small, square object, then
val would mostly follow
num_3. Note that this is a toy example, an in reality, there are many categorical features (several dozens) with a cardinality varying between 10 and 500. The number of
num_* columns is between 10 and 15.
We tried approaching this problem with boosted trees (using XGBoost) but it doesn't seem to generalise well, as trees are unable to learn a direct linear relationship as such (only mimic it with a high enough number of splits). Boosted linear models (such as XGBoost with
gblinear base model) do not seem to work a lot better.
What would be an appropriate choice of ML model for this kind of task?