Consider a simplified example where we are trying to predict some value val which happens to be equal to one of the num_* features. The task of the model is to select which of these num_* features to focus on, depending on the values of other features.
| color | size | shape | num_1 | num_2 | num_3 | num_4 | val |
|---|---|---|---|---|---|---|---|
| blue | large | rectangle | 0.3 | 0.6 | 0.0 | NaN | 0.6 |
| blue | large | rectangle | 0.7 | 0.2 | -0.3 | 0.1 | 0.2 |
| red | small | square | 0.1 | 0.1 | 0.3 | NaN | 0.3 |
| red | small | square | -0.4 | 0.5 | -0.5 | -0.5 | -0.5 |
| green | medium | round | -0.2 | 0.5 | 0.0 | 0.1 | -0.2 |
| blue | medium | oval | 0.7 | -0.3 | -0.1 | 0.5 | -0.5 |
What the model would learn in this toy example, is that when we have a blue, large, rectangle object, then val would mostly be equal to num_2, while when having a red, small, square object, then val would mostly follow num_3. Note that this is a toy example, an in reality, there are many categorical features (several dozens) with a cardinality varying between 10 and 500. The number of num_* columns is between 10 and 15.
We tried approaching this problem with boosted trees (using XGBoost) but it doesn't seem to generalise well, as trees are unable to learn a direct linear relationship as such (only mimic it with a high enough number of splits). Boosted linear models (such as XGBoost with gblinear base model) do not seem to work a lot better.
What would be an appropriate choice of ML model for this kind of task?
