0
$\begingroup$

I have a problem with a number of inputs and one binary output, which I have tried to train several classifiers to solve. Unfortunately, none of the classifiers (MLP, SVM, bagging) have achieved the required level of accuracy.

I am thinking of subdividing the problem by splitting the dataset into two, and using a different classifier on each half, or the same classifier but with different parameters (or the same parameters, but it will be learning a different problem).

One approach is to choose the attribute which has the highest correlation with the output, and split the data so that the lowest values are in one set, and the highest in another. I fond that one set gave very good accuracy, whereas the other had a lower accuracy than the combined data set.

Is there an alternative approach to subdividing the problem that anyone can suggest?

$\endgroup$
  • $\begingroup$ The purpose of bagging and random forests and most other approaches is so that you don't need to divide the data again to run the algorithms. And you did not explain why classification is of special interest as opposed to predicting risk. What is the ultimate problem you are trying to solve? $\endgroup$ – Frank Harrell Jul 7 '14 at 12:27
  • $\begingroup$ @Frank the classifier will bypass an existing algorithm which is computationally expensive. It is an engineering problem (I can't give any more details). I am getting accuracies which are in the 90s but they want a minimum of 98%. $\endgroup$ – John Jul 7 '14 at 12:38
  • $\begingroup$ I've often wondered if using a tree with some of it's variables coming from other ML techniques (outputs of an svm, logistic regression etc) might not work in such a situation. $\endgroup$ – aginensky Jul 7 '14 at 13:13
  • $\begingroup$ You still did not explain why you need arbitrary classifications as opposed to predicting probabilities of class membership. $\endgroup$ – Frank Harrell Jul 7 '14 at 13:29
  • $\begingroup$ @FrankHarrell because the engineering algorithm which this bypasses has a Boolean output $\endgroup$ – John Jul 7 '14 at 13:50
0
$\begingroup$

I have found that you just have to experiment by picking an attribute, splitting the data and then testing the results. If there is no real improvement, pick another attribute.

Like many things to do with neural networks, there probably isn't a set formula for it as using the correlation is just a heuristic.

In my example, I had 29 attributes. I split the data into two based on attribute which wasn't highly correlated to the output at all. The overall accuracy of the neural network improved from 88.3% to 97.7%.

$\endgroup$
  • $\begingroup$ The rationale for that strategy is unclear, and data splitting greatly reduces power unless the sample size is enormous. $\endgroup$ – Frank Harrell Jul 9 '14 at 11:37
  • $\begingroup$ What I didn't say @FrankHarrell is that I have an unlimited amount of training data. If you split the search space into two, then each neural network has a smaller domain, would this not tend to make it more accurate? That was my rationale. $\endgroup$ – John Jul 9 '14 at 13:05
  • $\begingroup$ The reverse is true. You need all the sample size you can get to fit models that are not pre-specified. $\endgroup$ – Frank Harrell Jul 9 '14 at 14:30
  • $\begingroup$ The sample size is not limited. By dividing the problem into two you are making the problem smaller. $\endgroup$ – John Jul 10 '14 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.