outcome <- c(1,0,0,1,1) string <- c('I love pasta','hello world', '1+1 = 2','pasta madness', 'pizza madness') df = df=data.frame(outcome,string) > df outcome string 1 1 I love pasta 2 0 hello world 3 0 1+1 = 2 4 1 pasta madness 5 1 pizza madness
Here I would like to use random forests to understand which words in the sentences contained in the
string variable are strong predictors of the
Someone suggested to use the
library(dplyr) library(tidyr) outcome <- c(1,0,0,1,1) string <- c('I love pasta','hello world', '1+1 = 2','pasta madness', 'pizza madness') df <- data.frame(outcome=factor(outcome,levels=c(0,1)),string, stringsAsFactors=FALSE) inp <- df %>% mutate(string=strsplit(string,split=" ")) %>% unnest(string) library(randomForest) mm <- model.matrix(outcome~string,inp) rf <- randomForest(mm, inp$outcome, importance=TRUE) imp <- importance(rf)
Problem is: my original dataset is much larger. Think of the
string column as a short english sentence of 5/6 words. The full dataset has 800k observations. Here, randomForest fails because of obvious memory issues.
What would be an alternative algorithm for my classification purpose?