1
$\begingroup$

I have dataset that has numeric, categorical, Continuous and Text data. I am using one classification model for numeric, categorical, Continuous and another for Text data. I get probabilities in both the cases. Now how do I combine these two models? Is this a valid approach given the two models use different parts of the training data?

Found a similar question here, but with no answers - Train Classifier on Text AND Categorical AND Numerical data

$\endgroup$

3 Answers 3

1
$\begingroup$

Is this a valid approach given the two models use different parts of the training data?

Training two classifiers on disjoint subsets of features you’ll not be able to capture the interaction between features belonging to different subsets, e.g. text and numerical ones.

vowpal wabbit can handle features of various types incl. numerical, categorical and text.

Edit: Just to make sure we are on the same page regarding the interactions in the context of your question.

Imagine the case depicted below, i.e. two classes are not perfectly separable in single dimension, but can be easily separated if you classifier consider both dimensions simultaneously.

You might have similar situation - the classes cannot be clearly separated by classifiers focusing only on a subset of features.

Source of figure: wikipedia

Example of classes non-separable in one dimension

$\endgroup$
1
  • $\begingroup$ Thanks for your reply. I do not have any such interactions between text and categorical/ numerical $\endgroup$ Dec 6, 2017 at 9:52
0
$\begingroup$

You can convert texts to vectors by tf idf https://en.m.wikipedia.org/wiki/Tf%E2%80%93idf?wprov=sfla1

Categorical to numerical by dummy https://en.wikipedia.org/wiki/Dummy_variable_%28statistics%29?wprov=sfla1

Both are supported by python and R.

After that, you have numerical values only.

Edit:

If you have two classifiers that are based on different approaches, then alternative options may work it out:

  • Take the LSTM on text as a first classifier in the boosting sequence. Subsequently, run the classification by boosting on categorical data.
  • If you have a strong motivation to use both classifiers, you can create an additional integrator that would have on inputs: (i) last states of the LSTM and (ii) results from your partial classifiers from boosting. These new inputs can be used for a new learning. All of them are numeric.
$\endgroup$
1
  • $\begingroup$ Thanks for your reply. I have two different models for these two sets of data. For text am using LSTM to classify and for rest of the data using xgboost. So now I have two models classifying the data. How can I club these two models? $\endgroup$ Nov 7, 2017 at 4:54
0
$\begingroup$

Now how do I combine these two models?

The simplest approach is to average the predictions. If it is a regression model, or model returns some kind of scores, you can simply average them between models. Additionally, for combining probabilities you have multiple options. If it returns classifications, then with more then two models you can use majority voting.

Is this a valid approach given the two models use different parts of the training data?

It is not only valid, but in many cases you could expect that this should improve the overall accuracy of the predictions. Combining different models is a popular approach, especially in machine learning competitions on sites like Kaggle. Usually you get better results if you combine models that differ from each other, then when using very similar models.

On another hand, in cases like yours, people often use models that use all the data, e.g. neural network that has a module for tabular data and another module for textual data, where the modules are combined at some point, to calculate final prediction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.