6
$\begingroup$

I've been getting inconsistent results with a binary classification problem I'm trying to solve using a linear classifier and a custom feature extraction pipeline, and decided to do a quick check of my code for bugs by training and testing my classifier on the same dataset. I expected this to yield a very high (100%?) accuracy/recall and precision stats, but to my surprise, I got results comparable to or even lower than the ones I normally get on distinct training and testing sets (~70% recall).

Should a classifier be very accurate when applied to its own training data, or do I just have a bug in my code? I'm not very experienced in ML so any help at all would be greatly appreciated! Thanks!!

Improve this question
$\endgroup$
  • $\begingroup$ For most classifiers (1-Nearest Neighbor aside), you shouldn't expect perfect performance on the the training data; 1-NN is trivially perfect on the training data. $\endgroup$ – Batman Jul 9 '16 at 1:12
4
$\begingroup$

No, your data may not be perfectly classifiable especially by a linear classifier and this is not always because of the classifier or the features you are using. None of the features may contain sufficent differences to provide a clear line.

You may try non-linear models which can provide better classification as well as higher risk of over-fitting. Using a validation set can help you identify whether you need a different model or the problem lies in the nature of your data.

Improve this answer
$\endgroup$
2
$\begingroup$

No, it's not always possible to create a linear boundary in the predictor space between all "1"s and "0"s in the data set (which is what would be required to have perfect linear classifier).

E.g., what if you had a single predictor and the training data were $y = (0,0,1,1)$, $x = (1,3,2,4)$. You can imagine similar scenarios with more predictors.

Improve this answer
$\endgroup$
1
$\begingroup$

No, every dataset is not linearly separable, as previous answers stated it.

Unless... You have more predictors than observations (or more columns than rows).

Therefore you should make sure that the feature extraction pipeline you use does not produce more features than your number of observations.

Improve this answer
$\endgroup$
  • $\begingroup$ That is also a very good point. $\endgroup$ – theGD Jul 8 '16 at 22:07
1
$\begingroup$

Imagine that it's truly a random data set. Let's say you're trying to classify the data into sick and healthy, and it just happens so that the incidence of sickness is truly random, at least independent of any of your predictors. In this case you shouldn't be getting good accuracy metrics without overfitting

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.