What are some valuable Statistical Analysis open source projects?

Question

What are some valuable Statistical Analysis open source projects available right now?

Edit: as pointed out by Sharpie, valuable could mean helping you get things done faster or more cheaply.

Answer 1

The R-project

http://www.r-project.org/

R is valuable and significant because it was the first widely-accepted Open-Source alternative to big-box packages. It's mature, well supported, and a standard within many scientific communities.

• Some reasons why it is useful and valuable
• There are some nice tutorials here.

Answer 2

For doing a variety of MCMC tasks in Python, there's PyMC, which I've gotten quite a bit of use out of. I haven't run across anything that I can do in BUGS that I can't do in PyMC, and the way you specify models and bring in data seems to be a lot more intuitive to me.

Answer 3

This may get downvoted to oblivion, but I happily used the Matlab clone Octave for many years. There are fairly good libraries in octave forge for generation of random variables from different distributions, statistical tests, etc, though clearly it is dwarfed by R. One possible advantage over R is that Matlab/octave is the lingua franca among numerical analysts, optimization researchers, and some subset of applied mathematicians (at least when I was in school), whereas nobody in my department, to my knowledge, used R. my loss. learn both if possible!

Answer 4

Two projects spring to mind:

1. Bugs - taking (some of) the pain out of Bayesian statistics. It allows the user to focus more on the model and a bit less on MCMC.
2. Bioconductor - perhaps the most popular statistical tool in Bioinformatics. I know it's a R repository, but there are a large number of people who want to learn R, just for Bioconductor. The number of packages available for cutting edge analysis, make it second to none.

Answer 5

Incanter is a Clojure-based, R-like platform (environment + libraries) for statistical computing and graphics.

Answer 6

Weka for data mining - contains many classification and clustering algorithms in Java.

Answer 7

ggobi "is an open source visualization program for exploring high-dimensional data."

Mat Kelcey has a good 5 minute intro to ggobi.

Answer 8

There are also those projects initiated by the FSF or redistributed under GNU General Public License, like:

• PSPP, which aims to be a free alternative to SPSS
• GRETL, mostly dedicated to regression and econometrics

There is even applications that were released just as a companion software for a textbook, like JMulTi, but are still in use by few people.

I am still playing with xlispstat, from time to time, although Lisp has been largely superseded by R (see Jan de Leeuw's overview on Lisp vs. R in the Journal of Statistical Software). Interestingly, one of the cofounders of the R language, Ross Ihaka, argued on the contrary that the future of statistical software is... Lisp: Back to the Future: Lisp as a Base for a Statistical Computing System. @Alex already pointed to the Clojure-based statistical environment Incanter, so maybe we will see a revival of Lisp-based software in the near future? :-)

Answer 9

RapidMiner for data and text mining

Answer 10

First of all let me tell you that in my opinion the best tool of all by far is R, which has tons of libraries and utilities I am not going to enumerate here.

Let me expand the discussion about weka

There is a library for R, which is called RWeka, which you can easily install in R, and use many of the functionalities from this great program along with the ones in R, let me give you a code example for doing a simple decision tree read from a standard database that comes with this package (it is also very easy to draw the resulting tree but I am going to let you do the research about how to do it, which is in the RWeka documentation:

library(RWeka)
iris <- read.arff(system.file("arff", "iris.arff", package = "RWeka"))
classifier <- IBk(class ~., data = iris)
summary(classifier)

There are also several python libraries for doing this (python is very very easy to learn)

First let me enumerate the packages you can use, I am not going to go in detail about them; Weka (yes you have a library for python), NLKT (the most famous open source package for textmining besides datamining), statPy, sickits, and scipy.

There is also orange which is excellent (I will also talk about it later), here is a code example for doing a tree from the data in the table cmpart1, that also performs 10 folds validation, you can also graph the tree

    import orange, orngMySQL, orngTree
    
    data = orange.ExampleTable("c:\\python26\\orange\\cmpart1.tab")
    
    domain=data.domain
    n=10
    buck=len(data)/n
    l2=[]
    for i in range(n):
        tmp=[]
        if i==n-1:
            tmp=data[n*buck:]
        else:
            tmp=data[buck*i:buck*(i+1)]
        l2.append(tmp)
    
    train=[]
    test=[]
    di={'yy':0,'yn':0,'ny':0,'nn':0}
    for i in range(n):
        train=[]
        test=[]
        for j in range(n):
            if j==i:
                test=l2[i]
            else:
                train.extend(l2[j])
        print "-----"
        trai=orange.Example(domain, train)
        tree = orngTree.TreeLearner(train)
        for ins in test:
            d1= ins.getclass()
            d2=tree(ins)
            print d1
            print d2
            ind=str(d1)+str(d2)
            di[ind]=di[ind]+1
    print di

To end with some other packages I used and found interesting

Orange:data visualization and analysis for novice and experts. Data mining through visual programming or Python scripting. Components for machine learning. Extensions for bioinformatics and text mining. (I personally recommend this, I used it a lot integrating it in python and it was excelent) I can send you some python code if you want me to.

ROSETTA: toolkit for analyzing tabular data within the framework of rough set theory. ROSETTA is designed to support the overall data mining and knowledge discovery process: From initial browsing and preprocessing of the data, via computation of minimal attribute sets and generation of if-then rules or descriptive patterns, to validation and analysis of the induced rules or patterns.(This I also enjoyed using very much)

KEEL:assess evolutionary algorithms for Data Mining problems including regression, classification, clustering, pattern mining and so on. It allows us to perform a complete analysis of any learning model in comparison to existing ones, including a statistical test module for comparison.

DataPlot: for scientific visualization, statistical analysis, and non-linear modeling. The target Dataplot user is the researcher and analyst engaged in the characterization, modeling, visualization, analysis, monitoring, and optimization of scientific and engineering processes.

Openstats: Includes A Statistics and Measurement Primer, Descriptive Statistics, Simple Comparisons, Analyses of Variance, Correlation, Multiple Regression, Interrupted Time Series, Multivariate Statistics, Non-Parametric Statistics, Measurement, Statistical Process Control, Financial Procedures, Neural Networks, Simulation.

Answer 11

Colin Gillespie mentioned BUGS, but a better option for Gibbs Sampling, etc, is JAGS.

If all you want to do is ARIMA, you can't beat X12-ARIMA, which is a gold-standard in the field and open source. It doesn't do real graphs (I use R to do that), but the diagnostics are a lesson on their own.

Venturing a bit farther afield to something I recently discovered and have just begun to learn...

ADMB (AD Model Builder), which does non-linear modeling based on the AUTODIF library, with MCMC and a few other features thrown in. It preprocesses and compiles the model down to a C++ executable and compiles it as a standalone app, which is supposed to be way faster than equivalent models implemented in R, MATLAB, etc. ADMB Project.

It started and is still most popular in the fisheries world, but looks quite interesting for other purposes. It does not have graphing or other features of R, and would most likely be used in conjunction with R.

If you want to work with Bayesian Networks in a GUI: SamIam is a nice tool. R has a couple of packages that also do this, but SamIam is very nice.

Answer 12

I really enjoy working with RooFit for easy proper fitting of signal and background distributions and TMVA for quick principal component analyses and modelling of multivariate problems with some standard tools (like genetic algorithms and neural networks, also does BDTs). They are both part of the ROOT C++ libraries which have a pretty heavy bias towards particle physics problems though.

Answer 13

GSL for those of you who wish to program in C / C++ is a valuable resource as it provides several routines for random generators, linear algebra etc. While GSL is primarily available for Linux there are also ports for Windows (See: this and this).

Answer 14

Few more on top of already mentioned:

• KNIME together with R, Python and Weka integration extensions for data mining
• Mondrian for quick EDA

And from spatial perspective:

• GeoDa for spatial EDA and clustering of areal data
• SaTScan for clustering of point data

Answer 15

I second that Jay. Why is R valuable? Here's a short list of reasons. http://www.inside-r.org/why-use-r. Also check out ggplot2 - a very nice graphics package for R. Some nice tutorials here.

Answer 16

This falls on the outer limits of 'statistical analysis', but Eureqa is a very user friendly program for data-mining nonlinear relationships in data via genetic programming. Eureqa is not as general purpose, but it does what it does fairly well, and the GUI is quite intuitive. It can also take advantage of the available computing power via the eureqa server.

Answer 17

Symbolic mathematics software can be a good support for statistics, too. Here are a few GPL ones I use from time to time:

1. sympy is python-based and very small, but can still do a lot: derivatives, integrals, symbolic sums, combinatorics, series expansions, tensor manipulations, etc. There is an R package to call it from R.
2. sage is python-based and HUGE! If sympy can't do what you want, try sage (but there is no native windows version).
3. maxima is lisp-based and very classical, intermediate in size between (1) and (2).

All three are in active development.

Answer 18

Meta.Numerics is a .NET library with good support for statistical analysis.

Unlike R (an S clone) and Octave (a Matlab clone), it does not have a "front end". It is more like GSL, in that it is a library that you link to when you are writing your own application that needs to do statistical analysis. C# and Visual Basic are more common programming languages than C/C++ for line-of-business apps, and Meta.Numerics has more extensive support for statistical constructs and tests than GSL.

Answer 19

• clusterPy for analytical regionalization or geospatial clustering
• PySal for spatial data analysis.