I have to build a logistic regression classifier to predict $\mathbf{y}$ given $\mathbf{x}$ where $\mathbf{x} \in \Re^{n}$ is an image and $\mathbf{y} \in \Re^{m}$ is a binary attribute vector (of $m$ attributes) with each $y_{i} \in [0,1]$.

The problem statement is that we have to build $m$ independent logistic regression classifiers to learn the mapping $\mathbf{x}\rightarrow y_{i}$ for $i = 1,2,..m$. I don't think this is a multi-label classification problem since the $m$ classifiers are learnt independently.

Kindly help me with the best strategy to solve the above problem. I can use sklearn and keras libraries only. There is no need for code; just let me know the best way to solve the above problem.


2 Answers 2


If we have already decided to use $m$ separate models, we train each model $i$ with data $(\mathbf{x}, y_i)$. Then, we feed new data $\mathbf{x}$ to $m$ models and concatenate their outputs as $\mathbf{y}'=(y'_1,..,y'_m).$

If we model this problem with $m$ separate models, we are assuming that given $\mathbf{x}$, $y_i$ is independent of $y_j$ for all $i$ and $j$. Nothing is wrong with this assumption and it could work. However, if this independence assumption does not work in practice, we could generally use a neural network with $n$ inputs, $m$ outputs, and loss function $\parallel \mathbf{y} - \mathbf{y'} \parallel^2$ for output $\mathbf{y'}$ and true attribute $\mathbf{y}$. We can use sigmoid function for the last layer to have $y'_i \in (0, 1)$. This way there is no independence assumption, and model $y'_i$ has shared parameters with $y'_j$. Of course, we can experiment with many design choices to see what network structure works best.

I don't think this is a multi-label classification problem since the $m$ classifiers are learnt independently.

If multiple $y_i$'s can be $1$ at the same time, you are right, it cannot be cast to a multi-class problem, since only one $y_i$ in $\mathbf{y}$ should be $1$ at a time, or at least $\sum_i y_i = constant$.


Your problem belongs to multi-label classification indeed. There is a distinction between multi-class (predict one of many labels) and multi-label (predict any of many labels).

What you think to do is called the Binary Relevance method in the multi-label literature, which comprises trying to train one classifier per label.

To use this method, you need to transform your labels when you train each of the classifiers: Make the positive class the examples belonging to the label you try to predict and all the other examples will be the negative class.

The easiest way to do this is to use the OneVsRestClassifier from Scikit-Learn with the MultiLabelBinarizer:




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.