I created a dataframe with 3 columns, feature_1, feature_2, and target, with the goal of having feature_1 and feature_2 predict the target. I standardized feature_1 and one-hot-encoded feature_2 (which has 100 categories and therefore creates 100 columns). I know for sure that feature_1 and target are correlated and so is feature_2 and target. The target is numbers from 1 to 10 and I create a correlated categorical (feature_2) variable by making a portion of the categorical values map perfectly with the target, for example, when target is a 3, set feature_2 to "A".
Creating a model with only feature_1 vs target I get val_loss of ~7.60. Creating a model with only feature_2 vs target I get a val_loss of ~8.40. In theory, I thought that combining the two features into one dataframe would yield a model with a better val_loss, but it does not. In fact, the val_loss is of the combined is ~8.00.
I tried a model which is an ensemble of model_1 and model_2 yet that ensemble's val_loss was 8.50, higher than all other models.
I'm not sure what to do or what the problem could be and I'm just looking for suggestions on what to try.
I am using Keras, pandas, and sklearn.