The usual method for adapting binary classifiers like various SVMs to multilabel data is one-vs-all, which assumes that labels are independent and in case of a prediction error we don't care what label the incorrect prediction should output.
But suppose I want to predict a score from 1 to 5, and I prefer to err closer to the truth, how do I go about it then? $\lt k$ vs $\ge k$ comes to mind, because in this case labels which are closer together will have larger training data overlap, but is it theoretically sound? Specifically, is overall performance likely to suffer? More generally, what are state-of-the-art techniques I could employ in this situation?