I have a large data set with over a million products. The NLP results look like this: this

A random row (reshaped) looks like this: this

The dataframe (image) contains information derived from the tokenization of a single product (" Dipperwell Faucet with Drain Stainless Steel Bowl Inner Overflow Cup").

Is it possible to predict the word "faucet" given the NLP output. The ML algorithms I am familiar with usually are in a format

Prediction~ Predictor 1+ Predictor 2.. etc..

In this case , the predictors are columns and the predicted value is a cell

  • $\begingroup$ "Is it possible"? I suppose so. Can you make your question more specific? $\endgroup$ – gung - Reinstate Monica Jun 8 '16 at 3:49
  • $\begingroup$ How do you recommend I do it? I will have to manually create a training set where the predicted variable is a token/ a combination of tokens. $\endgroup$ – Piyush Saxena Jun 10 '16 at 13:31
  • $\begingroup$ What is the NLP output? $\endgroup$ – Franck Dernoncourt Jun 26 '16 at 19:30
  • $\begingroup$ The output is tokens, POS and relationship between tokens . Thus, all categorical variables $\endgroup$ – Piyush Saxena Jun 28 '16 at 13:59
  • $\begingroup$ don't post images of data. hard to copy and paste and read $\endgroup$ – John H Oct 8 '18 at 2:19

I think you're trying to tag/classify your products into product "buckets" like faucets?

If so, the features you've proposed are ill-suited to a statistical approach. You should investigate word vectors as features. https://cs.stanford.edu/~quocle/paragraph_vector.pdf

You will still need to annotate a dataset, or provide "faucet" for each of some set observations for training. You could use the NLP techniques you're already employing to automate a portion of that work, effectively giving you a "suggestion" for you to review and edit if necessary.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.