# Predict multiple outcome vectors at once (no multinomial or multiclass)

Let's say I have a dataset where i need to predict 2 or more variables (classification). Please understand that i really don't mean multinomial or multiclass classification. It's all about predicting multiple outcome vectors.

$x_1$,$x_2$,$x_3$,$x_4$ $x_n$---> $y_1$,$y_2$

Does anyone know what sort of approach is good for this (preferably in r)? To be honest i've searched for answers a lot but i couldn't find anything.

• Why do you have to predict multiple classes at once? performance? are the outcomes even related? Jun 17, 2015 at 5:26
• Sounds like you should develop two separate models. If you're not interested in multinomial classification, there's no benefit to running them together Jun 15, 2018 at 23:20
• disagree with rsoren's comment. There are pros and cons to simultaneous Vs independent prediction (see my answer below). Oct 23, 2018 at 17:57

The decision you need to make, is whether you want to use two separate models, or one combined model. In general, using a single model is better, as it takes care of correlations between $$y_{1}$$ and $$y_{2}$$, but it's harder/requires you to think about the problem more.

For example, as far as I know, logistic regression doesn't lend itself very naturally to predicting two separate target variables. If both of your target variables are binary, you could make a single target variable which takes four outcomes, dealing with each of the four possible combinations This is probably a good approach, which deals very naturally with correlations in your two targets, but obviously as the cardinality of either of your targets increases, this doesn't scale very well.

Models that lend themselves quite naturally to multiple targets tend to be models that "generate features". Two very different classes of models that can be interpreted to do this are tree-based models (e.g. decision trees/random forests) and Deep Learning models.

If you train a decision tree with multiple categorical target variables, it essentially applies a compound condition to evaluate the best split at each node. For one variable, it looks for the partitioning of the feature space which best separates the data into two groups which give you the best (for example) Gini/Information Entropy separation of your target variable. With multiple targets, it looks for the split in feature space which gives you the best average (for example) Gini/Information Entropy split.

This has the advantage that adding more targets can actually make your process more robust to over-fitting, as the decision tree algorithm has to find a partitioning of your feature space which separates multiple target variables simultaneously. The more target variables a single partitioning of the data gives you information about, the less likely it is to be a fluke. The flip-side of this is that if your target variables are very different, it's perhaps not reasonable to expect that the same partitioning in space should give you information about both simultaneously.

Similarly, the hidden layers of Neural Networks take your raw features and try to learn a non-linear mapping to a new set of features which are the "right" features to feed to a logistic regression. A multiple-target Neural Network differs only from a standard neural network, in that it has to generate one non-linear mapping to find one set of features, that get passed to a separate logistic regression for each target. This is analogous to the decision tree discussed above. It's more robust to over-fitting, these "new" features that the neural network generates are less likely to be a fluke, if they can be used in multiple logistic regression models to get meaningful results over a range of targets, but likewise it might not be reasonable to think that very different target variables should be explainable with the same set features.

In conclusion, you have three options:

1. Use two separate models

Pros: Easy, allows you to learn features specific to each target

Cons: Assumes targets are independent of one and other so does not learn correlations between target variables

2. Create compound target from the two targets, categorical for every possible combination of the two

Pros: Best way to capture correlations between target variables

Cons: Can quickly get very sparse if either (or both) targets have high cardinality. Probably a good solution if you have enough data, but you'll need sufficient numbers of examples of every possible value of this compound target.

3: Train a model with two targets simultaneously (e.g. random forest or neural network)

Pros: Forces model to learn meaningful features and thus most robust to over-fitting. Code is easiest to keep track of as you have one model

Cons: If target variables are very different, you are likely to have much worse training loss than either of the other two suggests, as model as to compromise and learn one way to best determine two target variables.

Note that the 3rd method is the most robust to overfitting but is likely to have the worst training loss, so you'll need to cross-validate to determine which is best for your particular use case.

The notion of 'separate' models becomes confused depending on whether you are going forward or backward in a generative model. For instance, take the following model:

$$z_c \sim Bern(p)$$

$$y_i \sim N(\sum_{c=1}^C z_c \mu_c, \sigma^2)$$

If you are generating the data, the $$z_c$$ are independent Bernoulli rvs. However when you come to do inference you will find them to be tightly coupled to one another depending on the values of $$\{\mu_c\}$$.

For instance, if $$C=2, \sigma=1, \mu_1=0.9, \mu_2=1.1$$, then only one of $$z_1$$ or $$z_2$$ need be switched on to explain the data, and so they are not independent in the posterior.

Given this we realise that it is at best hopeful to think we can do good multi-label classification using totally separate models. Such an approach will work only to the extent that the labels are unrelated to one another.

It all depends on the situation. What is the number of features, and how big is your dataset? Do the labels influence each other? You could calculate the cross correlation matrix between the labels to find out the latter.

You could use regularized linear regression. It has the nice property that if you train it for one set of y's, you can immediately use the result to do the same for another set of y's, so training your classifier for multiple labels becomes very cheap.

If you have a small number of dimensions and a large number of training objects, you might want to use a kernelised classifier (in this case you might be able to find a decision boundary that is more descriptive than just a linear decision boundary). In this case you can use ridge / kernel regression (= kernelised regularized linear regression).

The performance of regularized linear regression (or ridge regression) is comparable to that of SVM's. SVM's are considered state of the art classifiers. If you don't have to worry about training times, I would just train multiple SVM's, one SVM for each label - it will only take longer. Depending on the number of features compared with the number of dimensions, you could look at multiple kernels.

Finally, the labels might influence each other as well, so the methods above might be suboptimal, because they do not take this into account. To improve on this naive baseline, check this wikipedia page: https://en.wikipedia.org/wiki/Multi-label_classification