0
$\begingroup$

I have a very large dataset that looks like

string                      x
this-is-a-nice-sentence     1
hello-my-name-bird          0
yay-this-is-awesome         1

Basically I want to understand what are the words that most likely predict x=1 (success).

How can I do that in Python? Thanks!

$\endgroup$
1
$\begingroup$

One approach is to create binary variables that indicates if the current sentence consists of a certain word. From your example, you can basically construct your training data in the following way:

  1. Combine all your sentences, make sure they are all small cases so that we don't duplicate count same word with capital letters. In your example, the unique words are:

    ["this","is","a","nice","sentence","hello","my","name","bird","yay","awesome"]

You can also remove stop words like "is" or "a" since these words are kind of not useful. You can find a list of stop words online.

  1. Construct the training data where each column is a word in your unique words list and for each row of your data (i.e. sentence), you check if that word is in this sentence, if yes, then label 1 for that column and 0 otherwise. Then for your label vector (y variable), you will simply have [1, 0, 1] in your case.

  2. Train your model, which ever model you like and see the performance and keep improving it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.