I have some questions regarding generalization and recall:

  1. It usually hear that a good classifier should "generalize" from the training data. Is that true?
  2. I feel(?) that a "general" classifier should improve recall, however a "specific" classifier should have more precision, is this true?
  3. If we think of a classifier that has to be learnt from a domain D1 and then used over a different domain D2, then a "generalized" classifier is preferred. In which other situations a "generalized" classifier is preferred?

Thanks in advance and sorry for the vague questions...

  • 1
    $\begingroup$ Out of the three points you made, only (1) makes sense. A good classifier that generalizes well, will have high recall and high precision. And there is no such thing as a generalized classifier. $\endgroup$
    – Pardis
    Jun 11 '12 at 5:45

"Generalization" refers to the ability of a classifier to correctly classify instances that it has not yet seen as part of its training and is always a desirable feature. The opposite of generalization would be over-fitting, which you always want to avoid (Although a classifier might also not generalize well if the training domain is different from the domain it will finally be applied to, which might not be due to over-fitting in the strictest sense).

The terms recall and precision have very specific meanings, while your use of "general" and "specific" are not standard terminology I don't think. For binary classifiers, it is customary to refer to one class of instances as "positive" and the other as "negative". For example, if you are trying to build a classifier that recognizes pictures with faces in them, then you might define the class of all pictures that contain faces as the positive class. The term recall is then defined as the proportion of all positive instances that were classified correctly, while precision is the proportion of all instances classified as positive that were actually positive.

Here is an example. Lets say of $40$ pictures, $30$ contain faces. If a classifier classifies $25$ pictures as positive, of which $20$ actually do contain faces, then the recall is $\frac{20}{30}=2/3$ while precision is $\frac{20}{25}=0.8$.

  • $\begingroup$ Thanks! Then, would it be correct to say that learning features "with high class discrimination power" improve the "generalization capability" of a classifier? Can we refer to such features as "general" features ? $\endgroup$ Jun 13 '12 at 5:42
  • 1
    $\begingroup$ using good features is always key to good classifier performance, but whether or not a specific feature helps in "generalizing" the classifier is not clear without context. You need to perform tests on the classifier to ensure that it works well in the intended domain and that it has not been overfitted to the training set. One way overfitting might occur is if you are choosing among a very large set of possible features. some features may seem to work very well in the training set, but this could be purely random noise. Therefore you need to test the generalizability of your classifier. $\endgroup$
    – ALiX
    Jun 14 '12 at 20:09
  • $\begingroup$ Thanks. I understand that the class discrimination power of a feature heavily depends on the context. However, In my case (I am dealing with sentiment analysis), some features like "happy" or "nice" (1) are likely to be better features than "map" or "phone"(2). So, my question is, although it all depends on the test dataset, if we prioritize features like (1) rather than (2) are we likely to end up with a classifier with higher recall ? $\endgroup$ Jun 15 '12 at 2:56
  • $\begingroup$ In short yes. this falls under the term "feature selection", which can be done manually, computationally, or both. Prior knowledge about your domain of interest should definitely help in creating better classifiers. But this can of course be tested as well. $\endgroup$
    – ALiX
    Jun 15 '12 at 3:47
  • $\begingroup$ Thanks again. I see. The problem I am facing now is target-independent sentiment analysis, which means that there is not a particular domain I am focusing on. This is why I am struggling now to get the most "general" sentiment features (the ones that could be more domain-independent than others) $\endgroup$ Jun 15 '12 at 5:42

Answering your 3rd question. Your training data should be as representative of the domain you are trying to learn as possible. The learned classifier should then be used in this same domain.

If you want to make use of the things you have learned in one domain and you think that this might be applied in another but somehow related domain then you are talking about 'transfer learning'. For example, a table classifier might have reusable parts if you want to learn a chair classifier.

  • $\begingroup$ Thanks for the name. Didn't know it was called "transfer learning" $\endgroup$ Jun 13 '12 at 5:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.