# Python as a statistics workbench

Lots of people use a main tool like Excel or another spreadsheet, SPSS, Stata, or R for their statistics needs. They might turn to some specific package for very special needs, but a lot of things can be done with a simple spreadsheet or a general stats package or stats programming environment.

I've always liked Python as a programming language, and for simple needs, it's easy to write a short program that calculates what I need. Matplotlib allows me to plot it.

Has anyone switched completely from, say R, to Python? R (or any other statistics package) has a lot of functionality specific to statistics, and it has data structures that allow you to think about the statistics you want to perform and less about the internal representation of your data. Python (or some other dynamic language) has the benefit of allowing me to program in a familiar, high-level language, and it lets me programmatically interact with real-world systems in which the data resides or from which I can take measurements. But I haven't found any Python package that would allow me to express things with "statistical terminology" – from simple descriptive statistics to more complicated multivariate methods.

What can you recommend if I wanted to use Python as a "statistics workbench" to replace R, SPSS, etc.?

What would I gain and lose, based on your experience?

-
FYI, there is a new python stats subreddit that is going off: reddit.com/r/pystats – naught101 Dec 4 '13 at 4:12

It's hard to ignore the wealth of statistical packages available in R/CRAN. That said, I spend a lot of time in Python land and would never dissuade anyone from having as much fun as I do. :) Here are some libraries/links you might find useful for statistical work.

• NumPy/Scipy You probably know about these already. But let me point out the Cookbook where you can read about many statistical facilities already available and the Example List which is a great reference for functions (including data manipulation and other operations). Another handy reference is John Cook's Distributions in Scipy.

• pandas This is a really nice library for working with statistical data -- tabular data, time series, panel data. Includes many builtin functions for data summaries, grouping/aggregation, pivoting. Also has a statistics/econometrics library.

• larry Labeled array that plays nice with NumPy. Provides statistical functions not present in NumPy and good for data manipulation.

• python-statlib A fairly recent effort which combined a number of scattered statistics libraries. Useful for basic and descriptive statistics if you're not using NumPy or pandas.

• statsmodels Statistical modeling: Linear models, GLMs, among others.

• scikits Statistical and scientific computing packages -- notably smoothing, optimization and machine learning.

• PyMC For your Bayesian/MCMC/hierarchical modeling needs. Highly recommended.

• PyMix Mixture models.

If speed becomes a problem, consider Theano -- used with good success by the deep learning people.

There's plenty of other stuff out there, but this is what I find the most useful along the lines you mentioned.

-
All answers were both helpful and useful, and would all deserve to be accepted. This one, however, does a very good job at answering the question: with Python, you have to put together lots of pieces to do what you want. These pointers will no doubt be very useful for anyone wanting to do statistics/modeling/etc. with Python. Thanks to everyone! – Fabian Fagerholm Aug 17 '10 at 18:42
@ars please do you know what is the best way to use Python with Windows ? – Stéphane Laurent Jul 24 '12 at 5:09
@StéphaneLaurent I usually install the various pieces myself, but for a quick start/install, you might consider: pythonxy. – ars Jul 24 '12 at 21:21
This script installs many of the libraries cited above: fonnesbeck.github.com/ScipySuperpack – Fr. Dec 8 '12 at 13:23
Pythonxy is nice but it can get annoying if you want to do large computations as it is only available for 32 bits. Here are unofficial binaries for installing many python packages. They can be quite useful if you decide to work under windows. lfd.uci.edu/~gohlke/pythonlibs @StéphaneLaurent – JEquihua Mar 26 '13 at 3:32

As a numerical platform and a substitute for MATLAB, Python reached maturity at least 2-3 years ago, and is now much better than MATLAB in many respects. I tried to switch to Python from R around that time, and failed miserably. There are just too many R packages I use on a daily basis that have no Python equivalent. The absence of ggplot2 is enough to be a showstopper, but there are many more. In addition to this, R has a better syntax for data analysis. Consider the following basic example:

Python:

results = sm.OLS(y, X).fit()


R:

results <- lm(y ~ x1 + x2 + x3, data=A)


What do you consider more expressive? In R, you can think in terms of variables, and can easily extend a model, to, say,

lm(y ~ x1 + x2 + x3 + x2:x3, data=A)


Compared to R, Python is a low-level language for model building.

If I had fewer requirements for advanced statistical functions and were already coding Python on a larger project, I would consider Python as a good candidate. I would consider it also when a bare-bone approach is needed, either because of speed limitations, or because R packages don't provide an edge.

For those doing relatively advanced Statistics right now, the answer is a no-brainer, and is no. In fact, I believe Python will limit the way you think about data analysis. It will take a few years and many man-year of efforts to produce the module replacements for the 100 essential R packages, and even then, Python will feel like a language on which data analysis capabilities have been bolted on. Since R has already captured the largest relative share of applied statisticians across several fields, I don't see this happening any time soon. Having said that, it's a free country, and I know people doing Statistics in APL and C.

-
+1 I just like this response because of the emphasis you put on R as a statistical language to work with data using formulaes and the like. That being said, I'm expecting a great positive impact of pandas (combined with statsmodels) in the Python community. – chl Oct 13 '11 at 21:44
in the Python community, patsy is addressing the need for "formula", which you describe, at times improving on what R offers: patsy.readthedocs.org/en/v0.1.0/index.html What's nice in Python is that all these aspects are handled orthogonally. Pandas will take care of timeseries and dataframe/series representation. patsy for the formulas. numpy for array representation and vectorization. statsmodels wraps statistics algos. scipy for optimization and a bunch of other stuff. The result is cleaner interfaces. R, in comparison, is more mature, but is a hairball. ../.. – blais Aug 13 '12 at 3:08
../.. I think in the long run the forces will push in the direction of more and more Python integration and you will find it will become quite a competitor to R. Cleaning data in R is such a PIA compared to Python, and it's never a trivial part of the job. – blais – chl Aug 13 '12 at 8:54
as an update: the first example in the statsmodels documentation is now results = smf.ols('Lottery ~ Literacy + np.log(Pop1831)', data=dat).fit(). Statsmodels is still far behind other statistical packages like R in terms of coverage, but there are more and more things you can do in python before you have to grab another language or statistical package. (GEE and Mixed will be in the next release.) – user333700 Apr 25 '14 at 15:26
"What's nice in Python is that all these aspects are handled orthogonally...". I disagree on a number of counts. There are significant overlap between numpy, scipy, statsmodels. R's design is much more modular and economic. Besides, most in not all of the conceptual innovations in data-oriented languages (not just formulas, but also data frames, a grammar of graphics, caret as a grammar of medels, knitr, and the still-developing grammar of data of dplyr) have originated in R. The Python community seems always a step behind, and overly focused on performance. – gappy Apr 25 '14 at 17:04

First, let me say I agree with John D Cook's answer: Python is not a Domain Specific Language like R, and accordingly, there is a lot more you'll be able to do with it further down the road. Of course, R being a DSL means that the latest algorithms published in JASA will almost certainly be in R. If you are doing mostly ad hoc work and want to experiment with the latest lasso regression technique, say, R is hard to beat. If you are doing more production analytical work, integrating with existing software and environments, and concerned about speed, extensibility and maintainability, Python will serve you much better.

Second, ars gave a great answer with good links. Here are a few more packages that I view as essential to analytical work in Python:

• matplotlib for beautiful, publication quality graphics.
• IPython for an enhanced, interactive Python console. Importantly, IPython provides a powerful framework for interactive, parallel computing in Python.
• Cython for easily writing C extensions in Python. This package lets you take a chunk of computationally intensive Python code and easily convert it to a C extension. You'll then be able to load the C extension like any other Python module but the code will run very fast since it is in C.
• PyIMSL Studio for a collection of hundreds of mathemaical and statistical algorithms that are thoroughly documented and supported. You can call the exact same algorithms from Python and C, with nearly the same API and you'll get the same results. Full disclosure: I work on this product, but I also use it a lot.
• xlrd for reading in Excel files easily.

If you want a more MATLAB-like interactive IDE/console, check out Spyder, or the PyDev plugin for Eclipse.

-
R is not a DSL in the usual sense of the term. It's a full, Turing complete programming language. – hadley Oct 13 '11 at 2:50
@hadley: Perhaps I am using "DSL" too colloquially, but for what it is worth, the Wikipedia page on DSLs explicitly lists S+ and R as examples of DSLs and Python as general purpose language. See en.wikipedia.org/wiki/Domain-specific_language. In the same vein, SAS is Turing-complete (only if the IML macro component is used), but I would hardly call it a complete language in a practical sense. I find R invaluable in my work, but I try to code using general purpose languages as much as possible rather than trying to do everything in R (or Excel for that matter). – Josh Hemann Oct 13 '11 at 16:12
I think it's unfair to include R and S in the same list as those other languages in wikipedia - there is nothing you can't do in R that you can do in python. Of course there are many things that are better suited to another programming language, but the same is true of Python. – hadley Oct 13 '11 at 16:36
Ahh, hadley removed R and S+ from the Wikipedia page the same day we exchanged comments, October 13, 2011. So, I often hear the mantra "R was developed by and for statisticians" as its foundational strength. Apparently, now it is a general purpose language, too... – Josh Hemann Oct 10 '12 at 4:32
Well, can I have multi-threaded code with R? Network asynchronous I/O? Believe me, these usecases actually arise in scientific computing. R is a DSL, in my opinion. It is strong at statistics, and bad at most other things. – Gael Varoquaux Aug 19 '14 at 21:15

I don't think there's any argument that the range of statistical packages in cran and Bioconductor far exceed anything on offer from other languages, however, that isn't the only thing to consider.

In my research, I use R when I can but sometimes R is just too slow. For example, a large MCMC run.

Recently, I combined python and C to tackle this problem. Brief summary: fitting a large stochastic population model with ~60 parameters and inferring around 150 latent states using MCMC.

1. Read in the data in python
2. Construct the C data structures in python using ctypes.
3. Using a python for loop, call the C functions that updated parameters and calculated the likelihood.

A quick calculation showed that the programme spent 95% in C functions. However, I didn't have to write painful C code to read in data or construct C data structures.

I know there's also rpy, where python can call R functions. This can be useful, but if you're "just" doing statistics then I would use R.

-
Inserting shameless plug for Rcpp :) – Dirk Eddelbuettel Aug 12 '10 at 14:37
curious if you've tried PyMC and how the performance compares (relative to python/C) for your models. – ars Aug 13 '10 at 7:31
@ars: In the case above, each iteration (of the 10^8 iterations) involved solving 5 ODEs. This really had to be done in C. The rest of the code was fairly simple and so the C code was straightforward. My application was non-standard and so PyMC wasn't applicable - also it was ~2 years ago. – csgillespie Aug 13 '10 at 12:33
@cgillespie: curiosity sated. sounds interesting, thanks. :) – ars Aug 13 '10 at 17:46

The following StackOverflow discussions might be useful

-
All of these discussions have been removed :-(. Perhaps this answer should be removed too? – Jonathan Mar 19 '13 at 15:47
That's sad. I've updated the links to refer to wayback machine copies. – Jeromy Anglim Mar 20 '13 at 5:54

I haven't seen the scikit-learn explicitly mentioned in the answers above. It's a Python package for machine learning in Python. It's fairly young but growing extremely rapidly (disclaimer: I am a scikit-learn developer). It's goals are to provide standard machine learning algorithmic tools in a unified interface with a focus on speed, and usability. As far as I know, you cannot find anything similar in Matlab. It's strong points are:

• High quality standard supervised learning (regression/classification) tools. Specifically:

• The ability to perform model selection by cross-validation using multiple CPUs

• Unsupervised learning to explore the data or do a first dimensionality reduction, that can easily be chained to supervised learning.

• Open source, BSD licensed. If you are not in a purely academic environment (I am in what would be a national lab in the state) this matters a lot as Matlab costs are then very high, and you might be thinking of deriving products from your work.

Matlab is a great tool, but in my own work, scipy+scikit-learn is starting to give me an edge on Matlab because Python does a better job with memory due to its view mechanism (and I have big data), and because the scikit-learn enables me to very easily compare different approaches.

-

One benefit of moving to Python is the possibility to do more work in one language. Python is a reasonable choice for number crunching, writing web sites, administrative scripting, etc. So if you do your statistics in Python, you wouldn't have to switch languages to do other programming tasks.

Update: On January 26, 2011 Microsoft Research announced Sho, a new Python-based environment for data analysis. I haven't had a chance to try it yet, but it sounds like an interesting possibility if want to run Python and also interact with .NET libraries.

-
I have done a lot of number crunching, one website and few administrative scripts in R and they are working quite nice. – mbq Aug 14 '10 at 19:40

Perhaps this answer is cheating, but it seems strange no one has mentioned the rpy project, which provides an interface between R and Python. You get a pythonic api to most of R's functionality while retaining the (I would argue nicer) syntax, data processing, and in some cases speed of Python. It's unlikely that Python will ever have as many bleeding edge stats tools as R, just because R is a dsl and the stats community is more invested in R than possibly any other language.

I see this as analogous to using an ORM to leverage the advantages of SQL, while letting Python be Python and SQL be SQL.

Other useful packages specifically for data structures include:

• pydataframe replicates a data.frame and can be used with rpy. Allows you to use R-like filtering and operations.
• pyTables Uses the fast hdf5 data type underneath, been around for ages
• h5py Also hdf5, but specifically aimed at interoperating with numpy
• pandas Another project that manages data.frame like data, works with rpy, pyTables and numpy
-
I've always find rpy sloppy to work with. It requieres large lines of codes with some simple functions, for example. – Néstor Jul 24 '12 at 2:48
Perhaps the rmagic extension for IPython (as pointed out by @CarlSmith) can make it easier to work with rpy2? See ipython.org/ipython-doc/dev/config/extensions/rmagic.html. – Jonathan Mar 19 '13 at 15:53

I am a biostatistician in what is essentially an R shop (~80 of folks use R as their primary tool). Still, I spend approximately 3/4 of my time working in Python. I attribute this primarily to the fact that my work involves Bayesian and machine learning approaches to statistical modeling. Python hits much closer to the performance/productivity sweet spot than does R, at least for statistical methods that are iterative or simulation-based. If I were performing ANOVAS, regressions and statistical tests, I'm sure I would primarily use R. Most of what I need, however, is not available as a canned R package.

-
+1 for distinguishing what area of statistics you work in. There are areas of statistical computing (e.g. unstructured text analysis and computer vision) that a lot of functionality exists for in Python, and Python is seemingly the lingua franca in those sub-domains. I think where the Python community has to catch up on is improving the data structures and semantics around classical statistical modeling that R's design is so good at. The scikits.statsmodels developers are making a lot of progress on that front: scikits.appspot.com/statsmodels – Josh Hemann Oct 13 '11 at 17:59

I would like to say that from the standpoint of someone who relies heavily on linear models for my statistical work, and love Python for other aspects of my job, I have been highly disappointed in Python as a platform for doing anything but fairly basic statistics.

I find R has much better support from the statistical community, much better implementation of linear models, and to be frank from the statistics side of things, even with excellent distributions like Enthought, Python feels a bit like the Wild West.

And unless you're working solo, the odds of you having collaborators who use Python for statistics, at this point, are pretty slim.

-
"I find it...". This answers forgets to say what "it" is, but I assume "it" is R. – Faheem Mitha Apr 5 '15 at 23:15
@FaheemMitha Fixed. – Fomite Apr 6 '15 at 18:07
Also, I think in " the odds ... is pretty slim" should be "the odds... are pretty slim". – Faheem Mitha Apr 6 '15 at 18:15

There's really no need to give up R for Python anyway. If you use IPython with a full stack, you have R, Octave and Cython extensions, so you can easily and cleanly use those languages within your IPython notebooks. You also have support for passing values between them and your Python namespace. You can output your data as plots, using matplotlib, and as properly rendered mathematical expressions. There are tons of other features, and you can do all this in your browser.

IPython has come a long way :)

-

What you are looking for is called Sage: http://www.sagemath.org/

It is an excellent online interface to a well-built combination of Python tools for mathematics.

-
The brilliant part about Sage is that it is essentially the union of a number of great free tools for mathematics, statistics, data analysis, etc. It is more than just Python; it has access to R, maxima, GLPK, GSL, and more. – shabbychef Oct 13 '11 at 16:48

Rpy2 - play with R stay in Python...

Further elaboration per Gung's request:

Rpy2 documentation can be found at http://rpy.sourceforge.net/rpy2/doc-dev/html/introduction.html

From the documentation, The high-level interface in rpy2 is designed to facilitate the use of R by Python programmers. R objects are exposed as instances of Python-implemented classes, with R functions as bound methods to those objects in a number of cases. This section also contains an introduction to graphics with R: trellis (lattice) plots as well as the grammar of graphics implemented in ggplot2 let one make complex and informative plots with little code written, while the underlying grid graphics allow all possible customization is outlined.

Why I like it:

I can process my data using the flexibility of python , turn it into a matrix using numpy or pandas and do the computation in R, and get back r objects to do post processing. I use econometrics and python simply will not have the bleeding edge stats tools of R. And R will unlikely ever be as flexible as python. This does require you to understand R. Fortunately, it has a nice developer community.

Rpy2 itself is well supported and the gentleman supporting it frequents the SO forums. Windows installation maybe a slight pain - http://stackoverflow.com/questions/5068760/bizzarre-issue-trying-to-make-rpy2-2-1-9-work-with-r-2-12-1-using-python-2-6-un?rq=1 might help.

-
Welcome to the site, @pythOnometrist. I suspect this is a helpful contribution. Would you mind giving a brief summary of Rpy2, so readers can decide if it's what they're looking for? – gung Apr 5 '13 at 21:57

I use Python for statistical analysis and forecasting. As mentioned by others above, Numpy and Matplotlib are good workhorses. I also use ReportLab for producing PDF output.

I'm currently looking at both Resolver and Pyspread which are Excel-like spreadsheet applications which are based on Python. Resolver is a commercial product but Pyspread is still open-source. (Apologies, I'm limited to only one link)

-
Again some interesting tools. I knew about Numpy, Matplotlib and ReportLab, but Pyspread seems like an interesting idea. At least I would like to type Python expressions in spreadsheet cells. While it doesn't solve all possible problems, it could be good for prototyping and playing around with data. – Fabian Fagerholm Aug 28 '10 at 6:21
+1 Wow python spreadsheets! Hadn't heard of those yet. I always wished OpenOffice/LibreOffice would really embrace and integrate python scripting in their spreadsheet software – User Oct 6 '11 at 19:34

great overview so far. I'm using python (specifically scipy + matplotlib) as a matlab replacement since 3 years working at University. I sometimes still go back because I'm familiar with specific libraries e.g. the matlab wavelet package is purely awesome.

I like the http://enthought.com/ python distribution. It's commercial, yet free for academic purposes and, as far as I know, completely open-source. As I'm working with a lot of students, before using enthought it was sometimes troublesome for them to install numpy, scipy, ipython etc. Enthought provides an installer for Windows, Linux and Mac.

Two other packages worth mentioning:

1. ipython (comes already with enthought) great advanced shell. a good intro is on showmedo http://showmedo.com/videotutorials/series?name=PythonIPythonSeries

2. nltk - the natural language toolkit http://www.nltk.org/ great package in case you want to do some statistics /machine learning on any corpus.

-

This is an interesting question, with some great answers.

You might find some useful discussion in a paper that I wrote with Roseline Bilina. The final version is here: http://www.enac.fr/recherche/leea/Steve%20Lawford/papers/python_paper_revised.pdf (it has since appeared, in almost this form, as "Python for Unified Research in Econometrics and Statistics", in Econometric Reviews (2012), 31(5), 558-591).

-

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Perhaps not directly related, but R has a nice GUI environment for interactive sessions (edit: on Mac/Windows). IPython is very good but for an environment closer to Matlab's you might try Spyder or IEP. I've had better luck of late using IEP, but Spyder looks more promising.

And the IEP site includes a brief comparison of related software: http://code.google.com/p/iep/wiki/Alternatives

-

I found a great intro to pandas here that I suggest checking out. Pandas is an amazing toolset and provides the high level data analysis capabilities of R with the extensive libraries and production quality of Python.

This blog post gives a great intro to Pandas from the perspective of a complete beginner:

http://manishamde.github.com/blog/2013/03/07/pandas-and-python-top-10/

-
Could you please write a few words about what qualities make it "great" so that readers can determine beforehand whether viewing it would be appropriate for them? – whuber Mar 16 '13 at 21:47
Sorry. Just realized I attached the wrong link in my original post. – padawan Mar 17 '13 at 18:15

I should add a shout-out for Sho, the numerical computing environment built on IronPython. I'm using it right now for the Stanford machine learning class and it's been really helpful. It's got built in linear algebra packages and charting capabilities. Being .Net it's easy to extend with C# or any other .Net language. I've found it much easier to get started with, being a windows user, than straight Python and NumPy.

-

No one has mentioned Orange before:

Data mining through visual programming or Python scripting. Components for machine learning. Add-ons for bioinformatics and text mining. Packed with features for data analytics.

I don't use it on daily basis, but it's a must-see for anyone who prefers GUI over command line interface.

Even if you prefer the latter, Orange is a good thing to be familiar with, since you can easily import pieces of Orange to your Python scripts in case you need some of its functionality.

-

Recent comparison from DataCamp provides clear picture about R and Python.

The usage of these two languages in the data analysis field. Python is used generally used when the data analysis tasks need to be integrated with web apps or if statistics code needs to be incorporated into a production database. R is mainly used when the data analysis tasks require standalone computing or analysis on individual servers.

I found it so useful in this blog and hope it would help others also to understand recent trends in both of these languages. Julia is also coming up in the area. Hope this helps !

-

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Note that SPSS Statistics has an integrated Python interface (also R). So you can write Python programs that use Statistics procedures and produce either the usual nicely formatted Statistics output or return results to your program for further processing. Or you can run Python programs in the Statistics command stream. You do still have to know the Statistics command language, but you can take advantage of all the data management, presentation output etc that Statistics provides as well as the procedures.

-

Python has a long way to go before it can be compared to R. It has significantly fewer packages than R and of lower quality. People who stick to the basics or rely only on their custom libraries could probably do their job exclusively in Python but if you're someone who needs more advanced quantitative solutions, I dare to say that nothing comes close to R out there.

It should be also noted that, to date, Python has no proper scientific Matlab-style IDE comparable to R-Studio (please don't say Spyder) and you need to work out everything on the console. Generally speaking, the whole Python experience requires a good amount of "geekness" that most people lack and don't care about.

Don't get me wrong, I love Python, it's actually my favourite language which, unlike R, is a real programming language. Still, when it comes to pure data analysis I am dependent to R, which is by far the most specialised and developed solution to date. I use Python when I need to combine data analysis with software engineering, e.g. create a tool which will perform automatisation on the methods that I first programmed in a dirty R script. In many occasions I use rpy2 to call R from Python because in the vast majority of cases R packages are so much better (or don't exist in Python at all). This way I try to get the best of both worlds.

I still use some Matlab for pure algorithm development since I love its mathematical-style syntax and speed.

-

When you need to move things around on the command line, pythonpy (https://github.com/Russell91/pythonpy) is a nice tool.

-

For those who have to work under Windows, Anaconda (https://store.continuum.io/cshop/anaconda/) really helps a lot. Installing packages under Windows was a headache. With Anaconda installed, you can set up a ready-to-use development environment with a one-liner.

For example, with

conda create -n stats_env python pip numpy scipy matplotlib pandas


all these packages will be fetched and installed automatically.

-

I believe Python is a superior workbench in my field. I do a lot of scraping, data wrangling, large data work, network analysis, Bayesian modeling, and simulations. All of these things typically need speed and flexibility so I find Python to work better than R in these cases. Here are a few things about Python that I like (some are mentioned above, other points are not):

-Cleaner syntax; more readable code. I believe Python to be a more modern and syntactically consistent language.

-Python has Notebook, Ipython, and other amazing tools for code sharing, collaboration, publishing.

-iPython's notebook enables one to use R in one's Python code so it is always possible to go back to R.

-Substantially faster without recourse to C. Using Cython, NUMBA, and other methods of C integration will put your code to speeds comparable to pure C. This, as far as I am aware, cannot be achieved in R.

-Pandas, Numpy, and Scipy blow standard R out of the water. Yes, there are a few things that R can do in a single line but takes Pandas 3 or 4. In general, however, Pandas can handle larger data sets, is easier to use, and provides incredible flexibility in regard to integration with other Python packages and methods.