Is it possible to have a model that aims to classify and regress at the same time?

For example if I have five independent variables and I want to use these 5 variables to predict the gender of the observation. Subsequently I let the predicted gender be the 6th independent variable and predict for income of the individual.

How I train the model is by splitting the dataset into two parts. The first part is used to train the model to predict gender, income is ignored here. The second part is used to train the model to predict the income of the individual based on all the other variables including gender.

Is this methodology sound? If not, why so?

  • 3
    $\begingroup$ Why does it have to be a single model? It makes much more sense to do the classification for gender first and do regression when you're done. If you do it this way you can use any algorithm you want and decouple unrelated issues. $\endgroup$ – Marc Claesen Nov 22 '13 at 9:25
  • $\begingroup$ @MarcClaesen has the answer here. I'm not sure if this is intentional, but this is essentially what is done in propensity analysis. The "propensity" of the treatment (gender in this case) is modeled and used in the outcomes regression model (income in this case). Thought now most propensity models use matching so its now exactly the same. $\endgroup$ – charles Nov 22 '13 at 20:45
  • $\begingroup$ Thanks Marc and charles for the clarification. I was actually afraid of dependency issues, which was why I split the dataset. But propensity analysis seems to describe my problem exactly and I will be looking into that. $\endgroup$ – user22119 Nov 23 '13 at 5:05
  • 1
    $\begingroup$ Not exactly similar to propensity score analysis. There, you would replace the five IVs by the propensity score and then use this score (the probability to be female based on those five IVs) along with actual sex to predict income. $\endgroup$ – Michael M Nov 23 '13 at 18:16
  • 1
    $\begingroup$ Almost 5 years later, the question receives 3 high quality answers haha :) $\endgroup$ – Firebug Aug 30 '18 at 18:54

Answering the specific question.

Yes, it is possible (but might be undesirable)

Directed graph models (such as ANNs) can cope with that.

You have 5 input variables, and you want to predict gender first, and include this prediction to predict income.

Basically, you need to connect all your inputs to an output, which is gender, then connect all your inputs again to a second output, income (a skip connection, in ANN terminology), plus your first output to the second. Also, don't forget bias terms.

enter image description here

Arrows are weights (coefficients) and circles are nodes.

Then, you want to minimize binomial loss on the prediction of gender, and (perhaps) squared loss on the prediction of income, optimizing the weights in the model. You can write this as your loss function.

$$\mathbb L_\text{total} (\text{input, gender, income})=\mathbb L _\text{binomial}(\text{gender})+\mathbb L _\text{squared}(\text{income})$$

You might want to normalize the loss terms, because, due to variance, one might dominate over the other.

Notice that the prediction of income depends on all weights in the model, so optimizing for the prediction of gender separately first might not be optimal to the predicition of income.


While this is specifically what you asked for, you have to ask if it's really useful. The separate prediction of gender and subsequently income including the gender predictions might be simpler, and above all, easier to implement.

Also, on imputation,

While studying these mediating relationships might be of interest, I have to warn you about the possibility of imputation.

If your objective is to account for missing gender information in some observations, then consider that the model won't use the actual gender information of the observations containing it. If the independent variables do not predict gender reasonably, you'll be basically forfeiting this information in your model, probably tending to a no informative output in the gender node.

  • 1
    $\begingroup$ Such a model as this network could easily be implemented with bayesian methodology in a language like bugs or stan $\endgroup$ – kjetil b halvorsen Aug 30 '18 at 14:41

Why do you want to do this? For your proposal to be possible and meaningful, it must be that you have some missing observations on the binary gender predictor (but not all). If that is the case, the usual approach is to use some (multiple) imputation method. See Multiple imputation for missing values or search this site!

(see also the comments for other ideas). There are some papers about imputation for binary (or categorical) predictors, for instance https://www.statalist.org/forums/forum/general-stata-discussion/general/29284-multiple-imputation-for-missing-categorical-variables, https://statisticalhorizons.com/ml-better-than-mi, https://www.sciencedirect.com/science/article/pii/S0167947310001490

(Sorry if I have misinterpreted the question, then it needs clarification)


As the other answers pointed out, this may not be something you want to do. But in cases where your 2 regression targets are closely related (say in the case of latitude and longitude) it may make sense.

One of the things I really love about neural networks is that they are very flexible, and allow you to define one model that does multiple things. For example, you can have one model that predicts both a classification target and a regression target.

Even if that's not what you should do on this dataset, I still think it's pretty cool, so I'm gonna share some code for a keras neural network in python that does classification and regression at the same time!

First, lets load some data. In this case, we're gonna predict sex (classification) and age (regression) from some census data:

import pandas as pd

adult = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data')
adult.columns = ['age', 'workclass', 'fnlwgt', 'edu', 'edu_num', 'marital_status',
                 'occupation', 'relationship', 'race', 'sex', 'capital_gain', 'capital_loss',
                 'hours_per_week', 'native_country', 'income']
adult["sex"] = adult["sex"].astype('category').cat.codes
adult["workclass"] = adult["workclass"].astype('category').cat.codes
adult["marital_status"] = adult["marital_status"].astype('category').cat.codes
adult["race"] = adult["race"].astype('category').cat.codes
adult["occupation"] = adult["occupation"].astype('category').cat.codes
adult["native_country"] = adult["native_country"].astype('category').cat.codes

target_bin = adult[['sex']].values.astype('float32')
target_num = adult[['age']].values.astype('float32')
X = adult[['workclass', 'edu_num', 'marital_status', 'occupation', 'race', 'capital_gain',
           'capital_loss', 'hours_per_week', 'native_country']].values.astype('float32')

X is our predictors, target_bin is the classification target, and target_num is the regression target.

Now lets define a keras model with one input (X) and 2 outputs (target_bin and target_num): from keras.layers import Input, Dense, BatchNormalization from keras.models import Model from keras.optimizers import Adam from keras.utils.vis_utils import plot_model

input = Input(shape=(X.shape1,), name='Input') hidden = Dense(64, name='Shared-Hidden-Layer', activation='relu')(input) hidden = BatchNormalization()(hidden) out_bin = Dense(1, name='Output-Bin', activation='sigmoid')(hidden) out_num = Dense(1, name='Output-Num', activation='linear')(hidden)

model = Model(input, [out_bin, out_num]) model.compile(optimizer=Adam(0.10), loss=['binary_crossentropy', 'mean_squared_error']) model.summary() plot_model(model, to_file='model.png')

Model Diagram

Now we can fit this model to our data:

model.fit(X, [target_bin, target_num], validation_split=.20, epochs=100, batch_size=2048)

We use a 20% validation set, where the model gets a logloss of 0.5649 and a mean squared error of 117.0967, which corresponds to an RMSE of 10.8. A logloss of <.69 means the model learned something useful for classification, and the RMSE of 10.8 is pretty good given that the mean of the numeric target is 38.6.

If you look at this model's predictions, you see it predicts 2 arrays. The first is the probability that the person is female, and the second is the person's predicted age.

You'll notice that I setup this network to use a shared hidden layer from the inputs to predict the outputs. You could also setup the network to e.g. use the sex prediction as a predictor for age, or other creative ways to relate the 2 variables.

So you can make one model to do both classification and regression!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.