0
$\begingroup$

I have a problem with regards to text classification/categorization. The task is bugging me for days already and as I am pretty new to AI and the field of natural language processing (NLP) I am just overwhelmed by the content online and available tools/libraries (e.g. NLTK, Keras, spaCy, etc.). It would be awesome if you could give me some guidance or clues on how you would approach the problem.

Issue: basicially I try to set up a tool for classifying text. I already have an extensive labeled dataset to work with. The input will always be a list of some sort (think of an Excel file with 500 rows). Each row contains a single word or a combination of words, i.e. no sentences.

A simplified example of my labeled dataset - input on the left, classification on the right:

  • "dog" -> "animal"
  • "dog owner" -> "person"
  • "dog owner house" -> "building"
  • "owner" -> "person"
  • "dog food" -> "food"
  • "food court" -> "building"

The dataset has around 2,000 of these classifications with in total 50 unique categories. How can I set up an algorithm that scans the input for example for the word "dog" - if it is only "dog" then it is the category "animal", if it is "dog" and "owner" it is the category "person", if it is "dog", "owner" and "house" it is the category "building" and so on.

If I set up a ton of if-else-statements as a decision tree it is just cumbersome and intransparent. Is there a way with NLP to solve such an issue?

Thank you very much in advance! Looking very much forward to your ideas and please let me know if I have to be more specific in any way.

Best regards, pythoneer

$\endgroup$
0
$\begingroup$

Take a look at the gensim python libary to use the word2vec function to create word embeddings. With that you can create vector representations for words. Similiar words have similiar vectors and are therefore closer to each other in the vector space. With that you can maybe do some kind of clustering or classificator.

$\endgroup$
1
  • $\begingroup$ Thank you, will give it a try! $\endgroup$
    – pythoneer
    Apr 14 '20 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.