1
$\begingroup$

You've got a supervised text-classification system that involves significant preprocessing of the source text (case-normalization, removing stop words, stemming, etc.) followed by some routine to produce a numerical document vectors for representing some language model (let's keep it simple and say BoW).

Your document vector matrix for train/test data:

| doc-id | "act" | "bat" | "cat" | ... | label |
|--------|-------|-------|-------|-----|-------|
|   d1   |   0   |   1   |   0   | ... | True  |
|   ...  |  ...  |  ...  |  ...  | ... | ...   |
|   dn   |   1   |   1   |   0   | ... | False |

You've trained, tested, iterated. You're getting good precision-recall. Now you're ready for production...

Are there common techniques, or any theory/conceptual approaches, ensuring that the production data produces features within the same feature space?

That is, how does the production system's preprocessing and vectorization steps produce new features (e.g. "after") that the system didn't produce during training/testing?

| doc-id | "act" | "after" | "bat" | "cat" | ... | label |
|--------|-------|---------|-------|-------|-----|
|   d1   |   0   |    1    |   0   |   0   | ... | True  |
|   ...  |  ...  |   ...   |  ...  |  ...  | ... |  ...  |
|   dn   |   0   |    1    |   0   |   1   | ... | False |

It is easy to imagine lower-level programmatic means of handling this: I'm interested in higher-level conceptual approaches, if they exist..

$\endgroup$

1 Answer 1

3
$\begingroup$

During production features like "after" are ignored because they were not seen during training. There is the concept of Pipeline (e.g., with scikit-learn at http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) for automating this process and avoiding creating different feature spaces. The whole idea however is that a model can not explicitly account (after training) for features that were not observed during training.

Training a classifier e.g., a Logistic Regression for the problem shown in the example actually means learning a set of weights that are associated with each feature. During prediction these weights are multiplied (dot product) with the value of the feature and depending on the value of the dot product a decision is made (True/False). In other words, training is the process of optimizing these associate weights that will result in better (more accurate) predictions. It follows that for features (like 'after') that are not seen during training no weights can be learned and, hence, they should be ignored during production as they can not contribute to the output of the aforementioned dot-product.

$\endgroup$
1
  • $\begingroup$ Yes. Understood RE unseen features. The pipeline concept is what I was looking for. Thanks. $\endgroup$
    – conner.xyz
    Dec 29, 2017 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.