0
$\begingroup$

I need to build a predictive model based on 20 classes. However, the constructed model achieved very low classification accuracy rate. So, I decided to re-group the classes to several of binary or multi-class classification problem based on the knowledges of the domain. So, I got like 8 of binary problems and 2 of three classes classification problem. However, some of the group/problems still have low accuracy rate, therefore, I have to recluster them again manually.

I am pretty new to the clustering scheme. Is there any algorithm or technique I can match them to the right group? I am thinking about writing a script to see the accuracy for every pair of each class. Please advise, thanks.

$\endgroup$
1
$\begingroup$

This is one of those core issues that will always bedevil those of us involved in quantitative and evidence-based decision-making -- the relative importance of judgement and gut intuition vs data-driven decisions and solutions.

For those invested more heavily in the pre-eminence of judgement over evidence such as the OP of this question, they will always be flummoxed when a statistical analysis or predictive model fails to confirm their intuitions -- "What happened?" is the inevitable question from these efforts.

Given this, my view is that you need to step back from trying to get statistics to confirm your judgement and let the data speak for itself. In other words, how did you get 20 classes in the first place? Who and why was this typology created? For what purpose?

Based on the OPs responses, it might be worth considering going back to the drawing board in terms of rebuilding the original set vs tinkering with the post-hoc product -- 20 classes may or may not be the right answer. This would permit the construction of a purely statistical grouping or schema. A critical step in this process is to ensure that training vs test samples are created as unsupervised learning methods can and will find statistically significant groupings on calibration data that fails to validate (or cross-validate) on external, o-o-s information.

Once a statistical typology is in place and validates o-o-s, then judgement has a role in identifying redundancies that overrule statistical significance in the groupings.

The statistical methods for achieving this are many. For instance, here's a paper that compares the classification accuracy of 100+ algorithms...

http://jmlr.csail.mit.edu/papers/volume15/delgado14a/delgado14a.pdf

$\endgroup$
  • $\begingroup$ Thanks for your advice, I really appreciate your answer. $\endgroup$ – Rapry Feb 21 '16 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.