I created a dataframe with 3 columns, feature_1, feature_2, and target, with the goal of having feature_1 and feature_2 predict the target. I standardized feature_1 and one-hot-encoded feature_2 (which has 100 categories and therefore creates 100 columns). I know for sure that feature_1 and target are correlated and so is feature_2 and target. The target is numbers from 1 to 10 and I create a correlated categorical (feature_2) variable by making a portion of the categorical values map perfectly with the target, for example, when target is a 3, set feature_2 to "A".

Creating a model with only feature_1 vs target I get val_loss of ~7.60. Creating a model with only feature_2 vs target I get a val_loss of ~8.40. In theory, I thought that combining the two features into one dataframe would yield a model with a better val_loss, but it does not. In fact, the val_loss is of the combined is ~8.00.

I tried a model which is an ensemble of model_1 and model_2 yet that ensemble's val_loss was 8.50, higher than all other models.

I'm not sure what to do or what the problem could be and I'm just looking for suggestions on what to try.

I am using Keras, pandas, and sklearn.


It is not always true that adding more features yields better results, even when these features are correlated with the target. The reason is that adding more features adds more complexity to your model, and more complex models are more likely to overfit to your training data. To avoid overfitting, you need to add more training data when making your model more complex.


First of all i agree with Hossein. Mixing two types of features not necessary leads to better classification results. I've just observed it myself few days ago, where i tried to mix character n-grams with content words for topic classification. Classification separatly on each feature type performed much better as on both together.

However, what i would suggest you is to construct an ensemble of classifiers $c_1, c_2, \ldots$, where each $c_i$ is trained on one sort of features (in my example $c_1$ on character n-grams and $c_2$ on content words), rather than using single-mixed feature vectors. Have a look on this article "Ensemble Machine Learning Algorithms in Python with scikit-learn".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.