Pandas / Statsmodel / Scikit-learn

Question

1. Are Pandas, Statsmodels and Scikit-learn different implementations of machine learning/statistical operations, or are these complementary to one another?

2. Which of these has the most comprehensive functionality?

3. Which one is actively developed and/or supported?

4. I have to implement logistic regression. Any suggestions as to which of these I should use?

Answer 1

1. Scikit-learn (sklearn) is the best choice for machine learning, out of the three listed. While Pandas and Statsmodels do contain some predictive learning algorithms, they are hidden/not production-ready yet. Often, as authors will work on different projects, the libraries are complimentary. For example, recently Pandas' Dataframes were integrated into Statsmodels. A relationship between sklearn and Pandas is not present (yet).

2. Define functionality. They all run. If you mean what is the most useful, then it depends on your application. I would definitely give Pandas a +1 here, as it has added a great new data structure to Python (dataframes). Pandas also probably has the best API.

3. They are all actively supported, though I would say Pandas has the best code base. Sklearn and Pandas are more active than Statsmodels.

4. The clear choice is Sklearn. It is easy and clear how to perform it.

   from sklearn.linear_models import LogisticRegression as LR
   logr = LR()
   logr.fit( X, Y )
   results = logr.predict( test_data)
   

Answer 2

I would like to qualify and clarify a bit the accepted answer.

The three packages are complementary to each other since they cover different areas, have different main objectives, or emphasize different areas in machine learning/statistics.

• pandas is mainly a package to handle and operate directly on data.
• scikit-learn is doing machine learning with emphasis on predictive modeling with often large and sparse data
• statsmodels is doing "traditional" statistics and econometrics, with much stronger emphasis on parameter estimation and (statistical) testing.

statsmodels has pandas as a dependency, pandas optionally uses statsmodels for some statistics. statsmodels is using patsy to provide a similar formula interface to the models as R.

There is some overlap in models between scikit-learn and statsmodels, but with different objectives. see for example The Two Cultures: statistics vs. machine learning?

some more about statsmodels

statsmodels has the lowest developement activity and longest release cycle of the three. statsmodels has many contributors but unfortunately still only two "maintainers" (I'm one of them.)

The core of statsmodels is "production ready": linear models, robust linear models, generalised linear models and discrete models have been around for several years and are verified against Stata and R. statsmodels also has a time series analysis part covering AR, ARMA and VAR (vector autoregressive) regression, which are not available in any other python package.

Some examples to show some specific differences between the machine learning approach in scikit-learn and the statistics and econometrics approach in statsmodels:

Simple linear Regression, OLS, has a large number of post-estimation analysis http://statsmodels.sourceforge.net/devel/generated/statsmodels.regression.linear_model.OLSResults.html including tests on parameters, outlier measures and specification tests http://statsmodels.sourceforge.net/devel/stats.html#residual-diagnostics-and-specification-tests

Logistic Regression can be done in statsmodels either as Logit model in discrete or as a family in generalized linear model (GLM). http://statsmodels.sourceforge.net/devel/glm.html#module-reference

GLM includes the usual families, discrete models contains besides Logit also Probit, multinomial and count regression.

Logit

Using Logit is as simple as this http://statsmodels.sourceforge.net/devel/examples/generated/example_discrete.html

>>> import statsmodels.api as sm
>>> x = sm.add_constant(data.exog, prepend=False)
>>> y = data.endog

>>> res1 = sm.Logit(y, x).fit()
Optimization terminated successfully.
         Current function value: 0.402801
         Iterations 7
>>> print res1.summary()
                           Logit Regression Results                           
==============================================================================
Dep. Variable:                      y   No. Observations:                   32
Model:                          Logit   Df Residuals:                       28
Method:                           MLE   Df Model:                            3
Date:                Sat, 26 Jan 2013   Pseudo R-squ.:                  0.3740
Time:                        07:34:59   Log-Likelihood:                -12.890
converged:                       True   LL-Null:                       -20.592
                                        LLR p-value:                  0.001502
==============================================================================
                 coef    std err          z      P>|z|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
x1             2.8261      1.263      2.238      0.025         0.351     5.301
x2             0.0952      0.142      0.672      0.501        -0.182     0.373
x3             2.3787      1.065      2.234      0.025         0.292     4.465
const        -13.0213      4.931     -2.641      0.008       -22.687    -3.356
==============================================================================
>>> dir(res1)
...
>>> res1.predict(x.mean(0))
0.25282026208742708