Consider a simplified example where we are trying to predict some value val which happens to be equal to one of the num_* features. The task of the model is to select which of these num_* features to focus on, depending on the values of other features.

color size shape num_1 num_2 num_3 num_4 val
blue large rectangle 0.3 0.6 0.0 NaN 0.6
blue large rectangle 0.7 0.2 -0.3 0.1 0.2
red small square 0.1 0.1 0.3 NaN 0.3
red small square -0.4 0.5 -0.5 -0.5 -0.5
green medium round -0.2 0.5 0.0 0.1 -0.2
blue medium oval 0.7 -0.3 -0.1 0.5 -0.5

What the model would learn in this toy example, is that when we have a blue, large, rectangle object, then val would mostly be equal to num_2, while when having a red, small, square object, then val would mostly follow num_3. Note that this is a toy example, an in reality, there are many categorical features (several dozens) with a cardinality varying between 10 and 500. The number of num_* columns is between 10 and 15.

We tried approaching this problem with boosted trees (using XGBoost) but it doesn't seem to generalise well, as trees are unable to learn a direct linear relationship as such (only mimic it with a high enough number of splits). Boosted linear models (such as XGBoost with gblinear base model) do not seem to work a lot better.

What would be an appropriate choice of ML model for this kind of task?


1 Answer 1


This is really a classification problem. Add a column to your data indicating which of the num_* features matches the target for each row, and train a classifier to predict that.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.