# How do R and Python complement each other in data science?

In many tutorials or manuals the narrative seems to imply that R and python coexist as complementary components of the analysis process. To my untrained eye, however, it seems that both languages sort of do the same thing.

So my question is if there are really specialized niches for the two languages or if it's just a personal preference whether to use one or the other?

• This really doesn't look like a duplicate of that thread. This is not about how to use Python to do statistics, it's about how Python can complement R (obviously R would be used to do the statistical analyses). – gung Oct 6 '16 at 12:24
• R if you were a statistician 10 years ago. Python if you want to put your code in production somewhere or reuse it. – djechlin Oct 6 '16 at 21:44
• – Juho Kokkala Oct 7 '16 at 17:01
• Owing to the popularity of this q., & in response to requests through flags, I've re-opened it but made it CW. – Scortchi Oct 7 '16 at 17:57
• What most answers tend to forget is that they are comparing the language Python to the implementation GNU-R. R also has other implementations (see Renjin, written in Java, Microsoft R Open, which is compiled with Intel MKL, FastR, pqR, etc). These try to correct some issues with GNU-R, a much more conservative implementation. Also, remember R has no formal definition. – Firebug Oct 13 '16 at 16:26

They are complementary. It is true that both can do the same things, yet this can be said of most languages. Each has its strengths and weaknesses. The common outlook seems to be that Python is best for data gathering and preparation, as well as for textual analysis. R is considered best for the data analysis, as it is a statistical language first and foremost.

R has a smorgasbord of packages for anything you can think of, but its staple is statistical analysis - from basic chi-square to factor analysis and hazard models, it is easy and robust. Some of the biggest names in statistics create R packages, and it has a lively community to help with your every need. ggplot2 is a standard in data visualization (graphs etc..). R is a vectorized language and built to loop through data efficiently. It also stores all data in the RAM, which is a double-edged sword - it is snappy on smaller data sets (although some might argue with me), but it can't handle big data well (although it has packages to bypass this, such as ff).

Python is considerably easier to learn than R - especially for those who have previous programming experience. R is just... weird. Python is great at data retrieval, and is the language to use for web scraping (with the amazing beautifulsoup). Python is known for its strength in string parsing and text manipulation. pandas is a great library for data manipulation, merging, transforming, etc., and is fast (and probably inspired by R).

Python is great when you need to do some programming. This is not surprising as it is a general-purpose language. R, however, with all its extensions, was built by statisticians for statisticians. So while Python may be easier and better and faster at many applications, R would be the go-to platform for statistical analysis.

• As someone who knows both, I agree with this; there may be lots of stylistic arguments, but the biggest difference is that once it's time to actually run causal statistical models Python is underdeveloped. The statsmodels package gets you some basics, but doesn't approach R, Stata or Sas. One notable exception is machine learning style analysis, for which Python's scikitlearn matches up well. For everything else I think good arguments can be made that Python is at least equal to R and often superior. Learning both is by no means a bad use of time. – Jeff Oct 6 '16 at 13:40
• I agree with most of this - but would emphasize issues more - R is not good for large data sets and doesn't handle sparse data sets (most libraries don't support it) very well (which is what many perform machine learning on) eg 'bag of words' type models where a factor/categorical variable's cardinality .might be in the 1000s. – seanv507 Oct 6 '16 at 14:24
• Not to start an argument in a comments thread, but the data.table package in R is specifically intended for efficient work around large data sets, and plugs in fine to distributed processing. – bright-star Oct 6 '16 at 18:57
• If you think that Python is better than R at data retrieval and string parsing then you've been using R very wrong the whole time. Same holds for the "R is not good for large data sets and doesn't handle sparse data sets (most libraries don't support it) very well". – gented Oct 7 '16 at 14:46
• adding to Trevor's boost of data.table; I'm also happy to say rvest is an EXCELLENT tool for handling most scraping tasks, and a lot easier to get up and running than beautifulsoup – MichaelChirico Oct 8 '16 at 2:34

I will try to formulate an answer touching the main points where the two languages come into play for data science / statistics / data analysis and the like, as someone who uses both.

The workflow in data analysis generally consists of the following steps:

1. Fetching the data from some sort of source (most likely a SQL/noSQL database or .csv files).
2. Parsing the data in a decent and reasonable format (data frame) so that one can do operations and think thereupon.
3. Applying some functions to the data (grouping, deleting, merging, renaming).
4. Applying some sort of model to the data (regression, clustering, a neural network or any other more or less complicated theory).
5. Deploying / presenting your results to a more-or-less technical audience.

## Fetching data

99% of the time, the process of fetching the data comes down to querying some sort of SQL or Impala database: both Python and R have specific clients or libraries that do the job in no time and equally well (RImpala, RmySQL for R and MySQLdb for Python work smoothly, not really much to add). When it comes to reading external .csv files, the data.table package for R provides the function fread that reads in huge and complicated .csv files with any custom parsing option in no time, and transforms the result directly into data frames with column names and row numbers.

## Organising the data frames

We want the data to be stored in some sort of table so that we can access any single entry, row or column with ease.

The R package data.table provides unbeatable ways to label, rename, delete and access the data. The standard syntax is very much SQL-like as dt[i, j, fun_by], where that is intended to be dt[where_condition, select_column, grouped_by (or the like)]; custom user-defined functions can be put in there as well as in the j clause, so that you are completely free to manipulate the data and apply any complicated or fancy function on groups or subsets (like take the i-th row, k-th element and sum it to the (k-2)-th element of the (i-1)-th row if and only if the standard deviation of the entire column is what-it-is, grouped by the last column altogether). Have a look at the benchmarks and at this other amazing question on SO. Sorting, deleting and re-naming of columns and rows do what they have to do, and the standard vectorised R methods apply, sapply, lapply, ifelse perform vectorised operations on columns and data frames altogether, without looping through each element (remember that whenever you are using loops in R you are doing it badly wrong).

Python's counterweapon is the pandas library. It finally provides the structure pd.DataFrame (that standard Python lacks, for some reason still unknown to me) that treats the data for what they are, namely frames of data (instead of some numpy array, numpy list, numpy matrix or whatever). Operations like grouping, re-naming, sorting and the like can be easily achieved and here, too, the user can apply any custom function to a grouped dataset or subset of the frame using Python apply or lambda. I personally dislike the grammar df[df.iloc(...)] to access the entries, but that is just personal taste and no problem at all. Benchmarks for grouping operations are still slightly worse than R data.table but unless you want to save 0.02 seconds for compilation there is no big difference in performance.

### Strings

The R way to treat strings is to use the stringr package that allows any text manipulation, anagram, regular expression, trailing white spaces or similar with ease. It can also be used in combination with JSON libraries that unpack JSON dictionaries and unlist their elements, so that one has a final data frame where the column names and the elements are what they have to be, without any non-UTF8 character or white space in there.

Python's Pandas .str. does the same job of playing with regular expressions, trailing or else as good as its competitor, so even here no big difference in taste.

## Applying models

Here is where, in my opinion, differences between the two languages arise.

R has, as of today, an unbeatable set of libraries that allow the user to essentially do anything they want in one to two lines of code. Standard functional or polynomial regressions are performed in one-liners and produce outputs whose coefficients are easily readable, accompanied by their corresponding confidence intervals and p-values distributions. Likewise for clustering, likewise for random forest models, likewise for dendograms, principal component analysis, singular value decompositions, logistic fits and many more. The output for each of the above most likely comes with a specific plotting class that generates visualisations of what you have just done, with colours and bubbles for coefficients and parameters. Hypotheses tests, statistical tests, Shapiro, Kruskal-Wallis or the like can be performed in one line of code by means of appropriate libraries.

Python is trying to keep up with SciPy and scikit-learn. Most of the standard analysis and models are available as well, but they are slightly longer to code and less-intuitive to read (in my opinion). More complicated machineries are missing, although some can be traced back to some combinations of the already existing libraries. One thing that I prefer doing in Python rather than in R is bag-of-word text analysis with bi-grams, tri-grams and higher orders.

## Presenting the results

Both languages have beautiful plotting tools, R ggplot2 above all and the corresponding Python equivalent. Not really much to compete, they do the job safe and sound, although I believe that if you are presenting the results you may have to use other tools—there are fancy colourful design tools out there and neither Python nor R are meant to astonish the audience with fancy red-and-green drag and drops. R has lately published a lot of improvements on its shiny app features, that basically allow it to produce interactive outputs. I never wanted to learn it, but I know it is there and people use it well.

## Side note

As a side note I would like to emphasise that the major difference between the two languages is that Python is a general purpose programming langauge, made by and for computer science, portability, deployments and so on and so forth. It is awesome at what it does and is straightforward to learn; there is nobody who does not like python. But it is a programming language to do programming.

R, on the other hand, was invented by and for mathematicians, physicists, statisticians and data scientists. If you come from that background everything makes perfect sense because it perfectly mirrors and reproduces the concepts used in statistics and mathematics. But if, instead, you come from a computer science background and want to simulate Java or C in R you are going to be disappointed; it does not have "objects" in the standard sense (well, it does, but not what one typically thinks they are...), it does not have classes in the standard sense (well, it does, but not what one typically thinks they are...), it does not have "pointers" or all other computer science structures - but just because it does not need them. Last but not the least: documentation and packages are straightforward to create and read (if you are using Rstudio); there is a large and passionate community out there, and it takes literally five seconds to Google "how to do insert-random-problem in R" whose first entry redirects you to a solution to the problem (done by someone else) with corresponding code, in no time.

Most industrial companies have their infrastructure built in Python (or a Python-friendly environment) that allows easy integration of Python code (just import myAnalysis anywhere and you are basically done). However, any modern technology or server or platform easily runs background R code without any problem as well.

• +6, this is a very good answer: detailed, accurate, and balanced. – gung Oct 8 '16 at 16:31
• +1 Best answer here. It's funny, I came to R from Java (albeit not in a direct route), and I've grown quite fond of it. I don't necessarily think that doing loops in R means you are doing it wrong. I have found myself scratching my head trying to circumvent using loops sometimes, but couldn't find a way around it (like creating new variable values in a data set based on conditions of previous rows and fetching data from secondary data sets). – Yuval Spiegler Oct 8 '16 at 18:43
• If only this plethora of R packages to do anything under the sun were all written by people who know how to do statistical computing/write reliable and robust numerical mathematical software/know numerical analysis, etc. Some of the R packages are very good. A very large number of R packages, even some written by famous authors, are total garbage - and many of the users don't even realize they're getting garbage. Oh, it's in CRAN, it must be good, or at least correct, or reliable ... WRONG!!!!! – Mark L. Stone Oct 10 '16 at 18:15
• "R has, as of today, an unbeatable set of libraries that allow the user to essentially do anything they want in one to two lines of code." I agree 100%, unless getting the correct answer is one of the things the user wants. – Mark L. Stone Oct 10 '16 at 18:27
• @MarkL.Stone Would you mind presenting an example of standard packages written by famous users that give wrong results? – gented Oct 10 '16 at 20:25
• Python is a general programming language: therefore, it is good for doing many other tasks in addition to data analysis. For example, if we want to automate our model execution in production server, then python is a really good choice. Other examples include connecting to hardware/sensors to read data, interacting with databases (relational or non-structured data like JSON), parsing data, network programming (TCP/IP), graphical user interface, interacting with shell, etc. (Well, why would a data scientist want to do so many of these kinds of task, which have little to do with predictive models? I think people have different definitions What is a data scientist? In some organizations, parsing the data and doing the descriptive analysis with dashboard is good enough for business and the data is not mature enough for doing predictive models. On the other hand, in many small companies, people may expect data scientists to do lots of software engineering. Knowing python will make you independent of other software engineers.)

• R has a lot of statistical packages that are much better than python or MATLAB. By using R, one can really think in model level instead of implementation detail level. This is a huge advantage in developing statistical models. For example, many people are manually implementing neural networks in python; doing such work may not help to understand why neural networks work, but just following the recipe to duplicate others' work to check if it works. If we are working in R, we can easily focus on the math behind the model, instead of implementation details.

In many cases, people use them together. Building software is easy to do in python, and building models is better in R. If we want to deliver a model in production but not a paper, we may need both. If your company has a lot of software engineers, you may need more R. And if your company has a lot of research scientists, you may need more python.

• Nice answer but you wrote 'Jason' instead of 'JSON', did give me a good chuckle – par Oct 6 '16 at 15:19
• I am not really a member of either the statistics or "machine-learning" communities, but I think the availability of advanced packages is somewhat community specific. I can believe that many new and state-of-the-art statistical techniques come with R code. But when I see computer vision/machine learning/neural net papers that come with ("high-level") codes ... they seem to be Matlab or python. – GeoMatt22 Oct 6 '16 at 22:11
• +1 for discussing how the languages influence the way one thinks. A massive advantage of R is, because it's built by statisticians for statisticians, how it encourages thinking in terms of modelling. Disclosure: I'm fluent in R, but only dabble in Python. – Ashe Oct 7 '16 at 13:35
• Um... what about scikit-learn? scikit-learn.org/stable – warship Oct 8 '16 at 20:59

Programmers of all stripes underestimate how much language choices are cultural. Web developers like Node.js. Scientists like Python. As a polyglot software engineer who can handle Javascript's fluidity and Java's rigidity all the same, I've realized there is not any intrinsic reason these languages are bad at each other's jobs -- just the enormous amount of packages, documentation, communities, books, etc. surrounding them.

(For intrinsic reasons one random language is better than some other language, see the forthcoming comments to this answer.)

My personal prediction is that Python is the way of the future because it can do everything R can - or rather, enough of what R can that dedicated programmers are working to fill in the gaps - and is a far better software engineering language. Software engineering is a discipline that deals with:

• trusting your code's reliability enough to put it in production (so any machine learning model that serves users in real time)
• ensuring your code can continue working as it undergoes modification and reuse (unit testing frameworks, for instance)
• a focus on readability, for the benefit of others, and of yourself in as little as 6 months
• a deep emphasis on code organization, for ease of versioning, backouts to previous working versions, and concurrent development by multiple parties
• preferring tools and technologies with better documentation, and ideally with the property that they won't work at all unless you use them right (this was my biggest gripe with Matlab -- I google a question and I have to read through their rather terrible forums searching for an answer)

Plus frankly Python is easier to learn.

Scientists and statisticians will realize they are stakeholders to good software engineering practice, not an independent and unbothered profession. Just my opinion, but papers proving the brittleness of academic code will support this.

This answer is all my opinion - but you asked a very opinionated question, and since it's well-received so far I felt you deserved an unpretentious, reasonably informed (I hope!) opinion in response. There's a serious argument for Python over R across the board and I would be remiss to try to post nonpartisan answer when reality may itself be partisan.

• Would'nt Julia, when it grows up, be a better alternative than Python? – kjetil b halvorsen Oct 6 '16 at 22:09
• @kjetilbhalvorsen "when it grows up" is more of an "if," it's very hard to supplant an established language, and it's basically a personal choice whether you want to be on frontier technology at the risk of it never being firmly adopted. IMO, Python has the momentum now. I'm not very familiar with Julia though so this is a pretty generic opinion though. – djechlin Oct 6 '16 at 22:34
• Except for the first bullet, I can't see how Python is inherently superior in any of the other four points. – Firebug Oct 7 '16 at 13:44
• "Python can do everything that R can" is false. "enough of what R can that dedicated programmers are working to fill in the gaps" well, that would be true for anything in the world - anything can be made to do anything as long as someone works on it. – gented Oct 7 '16 at 16:33
• @djechlin Totally not. Your example for reliability was "so any machine learning model that serves users in real time". Python is better at this because it supports deployment much more easily than R, and only that. Unit testing, readability, organization and documentation are not inherent qualities. You can implement your own tests, make your code easier to read, organize it and the R documentation is really easy to understand. – Firebug Oct 7 '16 at 17:27

I am an R user but I think Python is the future (I don't think it's the syntax)

Python is the future
The benefit of Python is as other people have already mentioned the much wider support, and, for programmers, more logical syntax.
Also the ability that you can translate findings from your analysis into a production system is much more straightforward.
Maybe it's because of Python being general purpose and R is not but even I raise my eyebrows when I see a productionized R pipeline.
But not only that, even for Advanced applications Python is quickly catching up (Scikit-learn, PyBrain, Tensorflow etc) and while R is still the lingua franca in academics on how to implement statistical methods Python has gotten huge in the professional sector due to the advent of advanced specialized libraries.

Many people seem to like to jump on the "R has bad syntax" bandwagon.
I wish to propose the syntax of R to be a good thing!
Assignment functions, lazy evaluation, non standard evaluation and formulas are huge benefits when using R.
It just saves so much time not to have to worry about escaping variable names referenced in your summary or how to construct the logic of what is modeled against what or looking at names with names() and then assigning new names by adding <- c("A", "B", "C").
When people complain about R's weird syntax they look at it as a programming language, not as a data science tool.
As someone coming from R and loving dplyr I find pandas' syntax a bit clumsy in comparison.
Yes it is a bit more flexible, but for most tasks you take a lot more keystrokes to perform a simple command than in R that are merely there to satisfy Python's parser, not to express your idea.

In summary
Of course it is wise to know both and while Python is getting there R's domain specific design choices just make it simpler for ad hoc work. The huge drawback of R is that it's difficult to leave its domain, which you basically have to do once you try to implement your findings in a sustainable way.

If you look at R as more of a statistical tool and not as a programming language, it is really great. It has far more flexibility than Stata or SPSS, but can do everything they can as well. I learned Stata during college, and R was easy enough to look at because I already had the perspective of the statistical tool and not a pure programming language experience that others might have.

I think frustration with R comes in to play when those who are programmers try to learn and understand R; but it is a great tool for those coming to R through a statistical background.

Python is great if you are already a great programmer; but for me as a beginner to programming and statistics just out of college, R was a much better choice. It is really just preference of which one fits your skillset and interests more.

In my experience, there's nothing easier than using R's dplyr + tidyr, ggplot and Rmarkdown in getting from raw data to presentable results. Python offers a lot, and I'm using it more and more, but I sure love the way Hadley's packages tie together.

Python has a wide adoption outside science, so you benefit from all that. As "An Angry Guide to R" points out, R was developed by a community, which had to the first order zero software developers.

I would say that today R has two main strengths: some really mature highly specialized packages in some areas, and state-of-the-art reproducible research package knitr.

Python appears to be better suited for everything else.

This is an opinion of course, as almost everything in this thread. I am kind of amazed that this thread is still alive.

As described in other answers, Python is a good general-purpose programming language, whereas R has serious flaws as a programming language but has a richer set of data-analysis libraries. In recent years, Python has been catching up to R with the development of mature data-analysis libraries such as scikit-learn, whereas R is never going to be fixed. In practice, I use Python (actually, Hy) for almost everything and only turn to R for relatively esoteric methods such as quantile regression (the implementation of which in Python's statsmodels appears to be broken). There are several ways to call R from Python; PypeR is one that's simple enough that I've gotten it to work in such hostile environments as a Windows server.