0
$\begingroup$

Beginner ML question here. I have a dataframe with a categorical column, a lot of the values are slightly different but essentially mean the same thing. Here's an example of such values:

Wireless Connection
Wireless Connection  Wi-Fi
Wireless Connection Issue 3g
Wireless Connection Issue Wi-Fi
Wireless/Connectivity

What would be the best way to condense/encode these values? Intuitively my thinking is to just find all instances of the values and convert them to single unique value like, Wireless Connection. For what it's worth, I'll be feeding the encoding of these values into some classifier, like a Decision Tree or Logistic Regression to predict if a customer's question in a call to customer service was answered.

Improve this question
$\endgroup$
4
  • $\begingroup$ Since your data is categorical, you don't have implicit relationships defined that would allow you to know that the meaning of values is similar. Would you be able to confirm that similar meaning could possibly be indicated by some sort of string similarity? $\endgroup$
    – mapto
    Commented May 14, 2019 at 13:30
  • 1
    $\begingroup$ Yes I was thinking of doing something like that; the total number of unique values is around 1,000 so my plan was just to go through them manually and make a judgment on them myself. $\endgroup$
    – Keith Johnson
    Commented May 14, 2019 at 14:07
  • 1
    $\begingroup$ Maybe human-validating them is a good idea. If new categories might emerge in future, you might consider semi-automating the process using libraries like TextDistance, jellyfish or something of the kind. Also consider different possible distance metrics, e.g. from this historical site. $\endgroup$
    – mapto
    Commented May 14, 2019 at 15:50
  • $\begingroup$ Another kind of idea is stats.stackexchange.com/questions/146907/… which collapses levels not from semantics, but if they give sufficiently similar predictions. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jun 26, 2019 at 23:50

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.