Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
In many tutorials or manuals the narrative seems to imply that R and python coexist as complementary components of the analysis process. To my untrained eye, however, it seems that both languages sort of do the same thing.
So my question is if there are really specialized niches for the two languages or if it's just a personal preference whether to use one or the other?
They are complementary. It is true that both can do the same things, yet this can be said of most languages. Each has its strengths and weaknesses. The common outlook seems to be that Python is best for data gathering and preparation, as well as for textual analysis. R is considered best for the data analysis, as it is a statistical language first and foremost.
R has a smorgasbord of packages for anything you can think of, but its staple is statistical analysis - from basic chi-square to factor analysis and hazard models, it is easy and robust. Some of the biggest names in statistics create R packages, and it has a lively community to help with your every need. ggplot2 is a standard in data visualization (graphs etc..). R is a vectorized language and built to loop through data efficiently. It also stores all data in the RAM, which is a double-edged sword - it is snappy on smaller data sets (although some might argue with me), but it can't handle big data well (although it has packages to bypass this, such as ff).
Python is considerably easier to learn than R - especially for those who have previous programming experience. R is just... weird. Python is great at data retrieval, and is the language to use for web scraping (with the amazing beautifulsoup). Python is known for its strength in string parsing and text manipulation. pandas is a great library for data manipulation, merging, transforming, etc., and is fast (and probably inspired by R).
Python is great when you need to do some programming. This is not surprising as it is a general-purpose language. R, however, with all its extensions, was built by statisticians for statisticians. So while Python may be easier and better and faster at many applications, R would be the go-to platform for statistical analysis.
data.table package in R is specifically intended for efficient work around large data sets, and plugs in fine to distributed processing.
$\endgroup$
– bright-star
Oct 6 '16 at 18:57
data.table; I'm also happy to say rvest is an EXCELLENT tool for handling most scraping tasks, and a lot easier to get up and running than beautifulsoup
$\endgroup$
– MichaelChirico
Oct 8 '16 at 2:34
I will try to formulate an answer touching the main points where the two languages come into play for data science / statistics / data analysis and the like, as someone who uses both.
The workflow in data analysis generally consists of the following steps:
99% of the time, the process of fetching the data comes down to querying some sort of SQL or Impala database: both Python and R have specific clients or libraries that do the job in no time and equally well (RImpala, RmySQL for R and MySQLdb for Python work smoothly, not really much to add). When it comes to reading external .csv files, the data.table package for R provides the function fread that reads in huge and complicated .csv files with any custom parsing option in no time, and transforms the result directly into data frames with column names and row numbers.
We want the data to be stored in some sort of table so that we can access any single entry, row or column with ease.
The R package data.table provides unbeatable ways to label, rename, delete and access the data. The standard syntax is very much SQL-like as dt[i, j, fun_by], where that is intended to be dt[where_condition, select_column, grouped_by (or the like)]; custom user-defined functions can be put in there as well as in the j clause, so that you are completely free to manipulate the data and apply any complicated or fancy function on groups or subsets (like take the i-th row, k-th element and sum it to the (k-2)-th element of the (i-1)-th row if and only if the standard deviation of the entire column is what-it-is, grouped by the last column altogether). Have a look at the benchmarks and at this other amazing question on SO. Sorting, deleting and re-naming of columns and rows do what they have to do, and the standard vectorised R methods apply, sapply, lapply, ifelse perform vectorised operations on columns and data frames altogether, without looping through each element (remember that whenever you are using loops in R you are doing it badly wrong).
Python's counterweapon is the pandas library. It finally provides the structure pd.DataFrame (that standard Python lacks, for some reason still unknown to me) that treats the data for what they are, namely frames of data (instead of some numpy array, numpy list, numpy matrix or whatever). Operations like grouping, re-naming, sorting and the like can be easily achieved and here, too, the user can apply any custom function to a grouped dataset or subset of the frame using Python apply or lambda. I personally dislike the grammar df[df.iloc(...)] to access the entries, but that is just personal taste and no problem at all. Benchmarks for grouping operations are still slightly worse than R data.table but unless you want to save 0.02 seconds for compilation there is no big difference in performance.
The R way to treat strings is to use the stringr package that allows any text manipulation, anagram, regular expression, trailing white spaces or similar with ease. It can also be used in combination with JSON libraries that unpack JSON dictionaries and unlist their elements, so that one has a final data frame where the column names and the elements are what they have to be, without any non-UTF8 character or white space in there.
Python's Pandas .str. does the same job of playing with regular expressions, trailing or else as good as its competitor, so even here no big difference in taste.
Here is where, in my opinion, differences between the two languages arise.
R has, as of today, an unbeatable set of libraries that allow the user to essentially do anything they want in one to two lines of code. Standard functional or polynomial regressions are performed in one-liners and produce outputs whose coefficients are easily readable, accompanied by their corresponding confidence intervals and p-values distributions. Likewise for clustering, likewise for random forest models, likewise for dendograms, principal component analysis, singular value decompositions, logistic fits and many more. The output for each of the above most likely comes with a specific plotting class that generates visualisations of what you have just done, with colours and bubbles for coefficients and parameters. Hypotheses tests, statistical tests, Shapiro, Kruskal-Wallis or the like can be performed in one line of code by means of appropriate libraries.
Python is trying to keep up with SciPy and scikit-learn. Most of the standard analysis and models are available as well, but they are slightly longer to code and less-intuitive to read (in my opinion). More complicated machineries are missing, although some can be traced back to some combinations of the already existing libraries. One thing that I prefer doing in Python rather than in R is bag-of-word text analysis with bi-grams, tri-grams and higher orders.
Both languages have beautiful plotting tools, R ggplot2 above all and the corresponding Python equivalent. Not really much to compete, they do the job safe and sound, although I believe that if you are presenting the results you may have to use other tools—there are fancy colourful design tools out there and neither Python nor R are meant to astonish the audience with fancy red-and-green drag and drops. R has lately published a lot of improvements on its shiny app features, that basically allow it to produce interactive outputs. I never wanted to learn it, but I know it is there and people use it well.
As a side note I would like to emphasise that the major difference between the two languages is that Python is a general purpose programming langauge, made by and for computer science, portability, deployments and so on and so forth. It is awesome at what it does and is straightforward to learn; there is nobody who does not like python. But it is a programming language to do programming.
R, on the other hand, was invented by and for mathematicians, physicists, statisticians and data scientists. If you come from that background everything makes perfect sense because it perfectly mirrors and reproduces the concepts used in statistics and mathematics. But if, instead, you come from a computer science background and want to simulate Java or C in R you are going to be disappointed; it does not have "objects" in the standard sense (well, it does, but not what one typically thinks they are...), it does not have classes in the standard sense (well, it does, but not what one typically thinks they are...), it does not have "pointers" or all other computer science structures - but just because it does not need them. Last but not the least: documentation and packages are straightforward to create and read (if you are using Rstudio); there is a large and passionate community out there, and it takes literally five seconds to Google "how to do insert-random-problem in R" whose first entry redirects you to a solution to the problem (done by someone else) with corresponding code, in no time.
Most industrial companies have their infrastructure built in Python (or a Python-friendly environment) that allows easy integration of Python code (just import myAnalysis anywhere and you are basically done). However, any modern technology or server or platform easily runs background R code without any problem as well.
Python is a general programming language: therefore, it is good for doing many other tasks in addition to data analysis. For example, if we want to automate our model execution in production server, then python is a really good choice. Other examples include connecting to hardware/sensors to read data, interacting with databases (relational or non-structured data like JSON), parsing data, network programming (TCP/IP), graphical user interface, interacting with shell, etc. (Well, why would a data scientist want to do so many of these kinds of task, which have little to do with predictive models? I think people have different definitions What is a data scientist? In some organizations, parsing the data and doing the descriptive analysis with dashboard is good enough for business and the data is not mature enough for doing predictive models. On the other hand, in many small companies, people may expect data scientists to do lots of software engineering. Knowing python will make you independent of other software engineers.)
R has a lot of statistical packages that are much better than python or MATLAB. By using R, one can really think in model level instead of implementation detail level. This is a huge advantage in developing statistical models. For example, many people are manually implementing neural networks in python; doing such work may not help to understand why neural networks work, but just following the recipe to duplicate others' work to check if it works. If we are working in R, we can easily focus on the math behind the model, instead of implementation details.
In many cases, people use them together. Building software is easy to do in python, and building models is better in R. If we want to deliver a model in production but not a paper, we may need both. If your company has a lot of software engineers, you may need more R. And if your company has a lot of research scientists, you may need more python.
scikit-learn? scikit-learn.org/stable
$\endgroup$
– warship
Oct 8 '16 at 20:59
Programmers of all stripes underestimate how much language choices are cultural. Web developers like Node.js. Scientists like Python. As a polyglot software engineer who can handle Javascript's fluidity and Java's rigidity all the same, I've realized there is not any intrinsic reason these languages are bad at each other's jobs -- just the enormous amount of packages, documentation, communities, books, etc. surrounding them.
(For intrinsic reasons one random language is better than some other language, see the forthcoming comments to this answer.)
My personal prediction is that Python is the way of the future because it can do everything R can - or rather, enough of what R can that dedicated programmers are working to fill in the gaps - and is a far better software engineering language. Software engineering is a discipline that deals with:
Plus frankly Python is easier to learn.
Scientists and statisticians will realize they are stakeholders to good software engineering practice, not an independent and unbothered profession. Just my opinion, but papers proving the brittleness of academic code will support this.
This answer is all my opinion - but you asked a very opinionated question, and since it's well-received so far I felt you deserved an unpretentious, reasonably informed (I hope!) opinion in response. There's a serious argument for Python over R across the board and I would be remiss to try to post nonpartisan answer when reality may itself be partisan.
I am an R user but I think Python is the future (I don't think it's the syntax)
Python is the future
The benefit of Python is as other people have already mentioned the much wider support, and, for programmers, more logical syntax.
Also the ability that you can translate findings from your analysis into a production system is much more straightforward.
Maybe it's because of Python being general purpose and R is not but even I raise my eyebrows when I see a productionized R pipeline.
But not only that, even for Advanced applications Python is quickly catching up (Scikit-learn, PyBrain, Tensorflow etc) and while R is still the lingua franca in academics on how to implement statistical methods Python has gotten huge in the professional sector due to the advent of advanced specialized libraries.
But R is not bad
Many people seem to like to jump on the "R has bad syntax" bandwagon.
I wish to propose the syntax of R to be a good thing!
Assignment functions, lazy evaluation, non standard evaluation and formulas are huge benefits when using R.
It just saves so much time not to have to worry about escaping variable names referenced in your summary or how to construct the logic of what is modeled against what or looking at names with names() and then assigning new names by adding <- c("A", "B", "C").
When people complain about R's weird syntax they look at it as a programming language, not as a data science tool.
As someone coming from R and loving dplyr I find pandas' syntax a bit clumsy in comparison.
Yes it is a bit more flexible, but for most tasks you take a lot more keystrokes to perform a simple command than in R that are merely there to satisfy Python's parser, not to express your idea.
In summary
Of course it is wise to know both and while Python is getting there R's domain specific design choices just make it simpler for ad hoc work. The huge drawback of R is that it's difficult to leave its domain, which you basically have to do once you try to implement your findings in a sustainable way.
If you look at R as more of a statistical tool and not as a programming language, it is really great. It has far more flexibility than Stata or SPSS, but can do everything they can as well. I learned Stata during college, and R was easy enough to look at because I already had the perspective of the statistical tool and not a pure programming language experience that others might have.
I think frustration with R comes in to play when those who are programmers try to learn and understand R; but it is a great tool for those coming to R through a statistical background.
Python is great if you are already a great programmer; but for me as a beginner to programming and statistics just out of college, R was a much better choice. It is really just preference of which one fits your skillset and interests more.
Adding to some of the prior answers:
In my experience, there's nothing easier than using R's dplyr + tidyr, ggplot and Rmarkdown in getting from raw data to presentable results. Python offers a lot, and I'm using it more and more, but I sure love the way Hadley's packages tie together.
Python has a wide adoption outside science, so you benefit from all that. As "An Angry Guide to R" points out, R was developed by a community, which had to the first order zero software developers.
I would say that today R has two main strengths: some really mature highly specialized packages in some areas, and state-of-the-art reproducible research package knitr.
Python appears to be better suited for everything else.
This is an opinion of course, as almost everything in this thread. I am kind of amazed that this thread is still alive.
As described in other answers, Python is a good general-purpose programming language, whereas R has serious flaws as a programming language but has a richer set of data-analysis libraries. In recent years, Python has been catching up to R with the development of mature data-analysis libraries such as scikit-learn, whereas R is never going to be fixed. In practice, I use Python (actually, Hy) for almost everything and only turn to R for relatively esoteric methods such as quantile regression (the implementation of which in Python's statsmodels appears to be broken). There are several ways to call R from Python; PypeR is one that's simple enough that I've gotten it to work in such hostile environments as a Windows server.
Edit: I encourage anybody who would like to argue about this further to talk to the authors of the linked essay instead of commenting on this answer.
