Lets say I have a set of unstructured text of student actions along with their grades. Is there a way if I want to know what words can lead to high grade or low grade?
For example, Data set
text,grade I was studying day by day, 80 I did my homework, 85 I missed alot of classes, 60 I stayed up late, 67
I expect that words with (study,homework) lead to high grade, while words with (missed,late) lead to low score
How can I achieve this?
I started with
- Pre processing text by removing stop words,punctuation, stemming and so on
- extracted ngrams features and used tf-idf as weighting function
- I ended up with a big data matrix for each word
I thought of using correlation to achieve what i want, but my matrix is very sparse and has alot of zeros