Python vs R for Text Mining Preprocessing [closed]

Question

I've been reading some articles on cleaning text data before doing text mining analysis on it.

I have experience in both Python and R and am wondering if one of these languages is an obviously better choice for cleaning text data i.e. stemming, lowercase, removing punctuation

Answer 1

I personally find the string parsing methods in Python much more intuitive than R, and chaining makes the code very readable.

R:

# Lowercase, remove !, tokenize

string<-"This is a string!!!"
newstring<-strsplit(tolower(gsub("!","",string))," ")

Python:

# Lowercase, remove !, tokenize

string="This is a string!!!"
newstring=string.lower().replace("!","").split()

Trimming leading and trailing whitespace in Python is as simple as string.strip(). In base R, you'd have to use regular expressions, though the stringr package will let you do it with str_trim(string).

Concatenating strings looks nicer in Python:

string="this " + "is " + "a " + "string"

vs R:

string=paste("this","is","a","string")

On the other hand, R has some handy functions to count words, e.g. table(newstring) vs. [newstring.count(x) for x in set(newstring)] in Python.

I much prefer R for actual data analysis though, in particular using the data.table package. However I often find myself doing pre-processing in Python and then importing into R for data analysis.

For more info, take a look at Python's builtin string methods and R's stringr package.

Answer 2

String processing in base Python and base R are both cumbersome (terrible). Since Jeff's answer above covers base R, I will cover string processing in the stringr package. This is a package designed to make string processing more consistent and more intuitive. There's almost no reason to ever use base R string processing. stringr lacks some functionality, but stringr is basically an elaborate wrapper package for the stringi package, which has tons of features. I'm not familiar with any Python equivalents to stringr or stringi, but a quick search found the regex package. This is apparently supposed to replace the re package, but it hasn't been done yet.

Concatenation

Easy string concatenation is still missing in base R and stringr, but one can add it by overloading the + operator with this function:

"+" = function(x, y) {

  if(is.character(x) || is.character(y)) {
    return(stringr::str_c(x, y))
  } else {
    .Primitive("+")(x, y)
  }
}

I've used this function for more than a year in my personal package and it doesn't break other things as far as I know. If you don't want to overload +, you can also use the %...% notation to create a regular infix function. Or you use use the already made %s+% function in stringi.

Base R

> paste0("a", "b", "c")
[1] "abc"

R with + overload

> "a" + "b" + "c"
[1] "abc"

Base Python

In[6]: "a" + "b" + "c"
Out[6]: 
'abc'

Python wins here since we have access to a built in infix operator as virtually every other language has too.

Splitting

Base R

> str = "This is a string!!!"
> strsplit(tolower(gsub("!","", str))," ")
[[1]]
[1] "this"   "is"     "a"      "string"

stringr R with pipe

> str = "This is a string!!!"
> str %>% str_to_lower() %>% str_replace_all("!", "") %>% str_split(" ")
[[1]]
[1] "this"   "is"     "a"      "string"

Here I used the pipe operator (%>%) to make the code easier to read. Read it as "then". So, first take the string, then make it lower case, then replace all the exclamation marks ("!") by nothing (""), and then split the string by space " ".

Base Python

In[7]: string="This is a string!!!"
In[8]: string.lower().replace("!", "").split()
Out[8]: 
['this', 'is', 'a', 'string']

Python uses methods to get this task done while R uses functions. I prefer functional programming, but you may not.

Extracting regex matched groups

Base R

> str = "xy1234wz98xy567"
> r = "xy(\\d+)"
> gsub(r, "\\1", regmatches(str, gregexpr(r, str))[[1]])
[1] "1234" "567" 

stringr R

> str = "xy1234wz98xy567"
> str_match_all(str, "xy(\\d+)")[[1]][, 2]
[1] "1234" "567" 

The reason for the subsetting at the end is that str_match_all returns a list of matrices. Each matrix corresponds to one regex pattern; one can give multiple since the function is vectorized. Each matrix has the results in columns. The rows are results for different input strings because the function is also vectorized over input strings. The first column has the complete match and the follow columns have the matched groups. So in our case, there's only 1 pattern and only 1 group, so we subset to the first matrix and fetch the second column.

Base Python

In[40]: string = "xy1234wz98xy567"
In[41]: re_pattern = re.compile("xy(\d+)")
In[42]: re_pattern.findall(string)
Out[42]: 
['1234', '567']

In general stringr is better than Python's re package for regex tasks. Hopefully someone will port the stringr package to Python at some point just as was done with data frames in pandas.

Answer 3

Python is more robust for production use cases, but is somewhat less friendly than R for tinkering around data. Pandas gets most of the way there, but is fundamentally a DSL on top of Python with resultant inconsistencies that R simply doesn't have. Mainly that's because R is vectorized throughout, whereas python is fundamentally a scalar language with vector extensions through Numpy and Pandas. Whenever I have "dirty data" I always reach for R, because it is intrinsically designed around data workflows. R is also, in my opinion, vastly superior to Python for static plotting.

That said, Python has some enormous momentum in areas other than statistics, which means that it often plugs in better to text data sources than R. Think all the web scraping tech that happens in Python, the PDF parsers, the twitter clients, not to mention Python is the de-facto language of machine learning. R usually has the same but less developed. Moreover, Python has threads which, though much maligned because of the GIL, do give you more freedom as a programmer to do stuff concurrently.

Python is also significantly faster than R, mainly because its memory management is much better. You're talking between 3 and 10 times faster in my experience, though we're still slow by comparison with the compiled languages.

R has the advantage of huge libraries that are sometimes easier to install and use. Interestingly, outside of machine learning, academia seems to do a lot of work on cutting edge algorithms in R, and that might be for historical reasons but I suspect is more because it is an (even) better "playground" for data exploration.

I think it's fair to say that R feels like a higher level, functional language, by comparison to the (still high level) Python. That means that if you're doing what it's designed for (data/stats/plotting), it's unbeatable. But the tradeoff is that when you step outside these domains, pain comes quickly.