1
$\begingroup$

I'm developing a classification model and for each sample I have a mix of numeric and categorical features. I also have a paragraph of text describing each sample. I'm looking for ways to incorporate this text data into the classification model.

For example lets say I would like to predict what category a car falls into using its features.

I could have features such as number of doors, horse power and weight. But I also have a paragraph of unstructured text describing each car. intuitively there are many key words and concepts in the text that can be used to classify the car. How can this information be used in a classification algorithm?

I know one option would be to extract key words and encode them as (1,0) if they appear or don't appear for that sample. But the issue is that due to the large amount of variation and potential keywords it would drastically increase the feature dimension.

I am also aware of text classification models but these only use the text and would omit the other numeric and categorical features.

Is there a modeling method for using text as a feature (or extracting features or expressing text as an embedding/vector) so that It can be used in a classification algorithm such as a decision tree or XGboost.

$\endgroup$
1
$\begingroup$

Extracting keywords as binary features were the state of the art for a very long time. Most implementations of decision trees/forests can deal with a pretty large set of features. You should also consider weighting the features with TF-IDF scores. If speed is really critical, there are toolkits for linear models that can easily deal with a large number of sparse features (such Vowpal Wabbit).

Dense representation from neural networks will work also with traditional ML algorithms. From your question, it seems to me that you don't want to run heavy-weighted neural representations models such as BERT (which would certainly work here too). In that case, a reasonable thing would be representing the paragraphs using an average word embedding. In depends on the length of the paragraphs, but it may pay off to extract keywords first and then do the averaging.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.