I have a problem with continuous feature and outcome data. The features are weak predictors. I'd like to be able to cluster my features into $k$ classes. This is not semi-supervised learning so much as poorly-supervised learning---I want to cluster the feature data, with some influence from the known outcomes in the training data, but not take it as gospel.
Is there a nice way to cluster while incorporating the outcome training data in a single step?
I can think of regressing the data and then clustering the predicted outcomes (each a scalar); however, the regression is not very good so I don't want to rely on it heavily. I can also cluster the data unsupervised ($k$-means) and then rank them by the mean outcome of each cluster in the training data. The latter works surprisingly well, but doesn't taken into account the training data that I have available.