0
$\begingroup$

We had a survey of >1500 patients, and we did cluster analysis, and grouped them into 3 clusters. We want to develop a algorithm to predict cluster membership of future patients. But the question is: can we use the cluster membership obtained from cluster analysis as a target in future prediction?

To me, the target for classification prediction should have real data in the training set, isn't it?

Any comments will be appreciated. Thanks.

$\endgroup$
2
  • $\begingroup$ real data in what sense? Many groups "real" out there in the population are actually artificial groups "clustered" through human+statistical decisions. Such as groups based on inventories/interviews. $\endgroup$
    – ttnphns
    May 25, 2016 at 16:35
  • $\begingroup$ By "read data", I meant the target is a clearly defined and observed, not estimated from cluster analysis. For example, an email is spam or not, or a tumor is benign or not, etc, which all have a clear and definite answer. But the cluster membership, is not from observed, and may not be correct, especially for those clusters overlapping more or less. $\endgroup$
    – ziweiguan
    May 25, 2016 at 21:24

1 Answer 1

2
$\begingroup$

But the question is: can we use the cluster membership obtained from cluster analysis as a target in future prediction?

Sure, you absolutely can. The "risks" involved here are (1) whether or not the classification model is any good at picking up cluster memberships and (2) whether the cluster memberships are measuring anything relevant to the problem that you're trying to solve and (3) the "failure modes" of the cluster analysis.

By "failure modes" I mean knowing what happens when a new, previously undiscovered cluster appears in your new data. For example, you might have had a two or three observations which were quite far from your clusters and that the clustering algorithm ignored because they were "noise". But if you collect more data, perhaps you find that you have an additional cluster in the vicinity of those two or three points. What your classifier does depends on the model, and whether this matters at all depends on your particular context.

If you're confident that you can address these concerns, then I'd say you're in the clear!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.