1
$\begingroup$

I have training data with ~600K instances. If I split the training data into four segments and build four separate classifiers, I get much higher accuracy for each model than if I train a single model over the entire data set.

I can do the split for this specific data set because I have prior knowledge of the data. I am wondering if there is an algorithm that will automatically do this task for me.

Given a labelled training data set, divide it into k subsets for k classifiers that are likely to result in higher accuracy.

So something like random forest, but without the "randomness"

$\endgroup$
  • $\begingroup$ When you say "train a model", which algorithm(s) are you using? Did you use k-fold cross-validation to assess model accuracy? $\endgroup$ – Tchotchke Sep 24 '15 at 12:51
  • $\begingroup$ we used SVM and decision tree and a 10-fold cross validation $\endgroup$ – user5371499 Sep 24 '15 at 13:07
1
$\begingroup$

One way of approaching this issue is to try ensemble methods such as bagging, boosting. These approaches generally perform better than single model. This paper would be a good starting point.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

The apparent performance of a method will go up as you decrease the sample size, because the signal:noise ratio (amount of overfitting) gets worse. But you have an additional problem in using an improper accuracy scoring rule, which is akin to adding a lot of meaningless noise to the accuracy score no matter how large the sample is. Using an improper accuracy score for judging the relative merits of different models will lead to incorrect judgements a significant portion of the time.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The accuracy also varies for the same-sized samples. For example, if I take 150K random instances from the original data and train model A, and take 150K instances (that I know to be 'different') and train model B, then miss classification rate is much higher for A (10-fold cross validation) $\endgroup$ – user5371499 Sep 24 '15 at 13:18
0
$\begingroup$

If you are looking to automate the clustering, you could use a k-means or hierarchical clustering approach.

If you want an algorithm that would take your clusters into account, a non-linear approach such as boosting or random forest would probably work well. If the clusters you've identified are important, the trees should split along those features and then you don't have to manually specify the clusters.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.