What are some valuable Statistical Analysis open source projects available right now?
Edit: as pointed out by Sharpie, valuable could mean helping you get things done faster or more cheaply.
What are some valuable Statistical Analysis open source projects available right now?
Edit: as pointed out by Sharpie, valuable could mean helping you get things done faster or more cheaply.
The R-project
R is valuable and significant because it was the first widely-accepted Open-Source alternative to big-box packages. It's mature, well supported, and a standard within many scientific communities.
For doing a variety of MCMC tasks in Python, there's PyMC, which I've gotten quite a bit of use out of. I haven't run across anything that I can do in BUGS that I can't do in PyMC, and the way you specify models and bring in data seems to be a lot more intuitive to me.
This may get downvoted to oblivion, but I happily used the Matlab clone Octave for many years. There are fairly good libraries in octave forge for generation of random variables from different distributions, statistical tests, etc, though clearly it is dwarfed by R. One possible advantage over R is that Matlab/octave is the lingua franca among numerical analysts, optimization researchers, and some subset of applied mathematicians (at least when I was in school), whereas nobody in my department, to my knowledge, used R. my loss. learn both if possible!
Two projects spring to mind:
Incanter is a Clojure-based, R-like platform (environment + libraries) for statistical computing and graphics.
Weka for data mining - contains many classification and clustering algorithms in Java.
ggobi "is an open source visualization program for exploring high-dimensional data."
Mat Kelcey has a good 5 minute intro to ggobi.
There are also those projects initiated by the FSF or redistributed under GNU General Public License, like:
There is even applications that were released just as a companion software for a textbook, like JMulTi, but are still in use by few people.
I am still playing with xlispstat, from time to time, although Lisp has been largely superseded by R (see Jan de Leeuw's overview on Lisp vs. R in the Journal of Statistical Software). Interestingly, one of the cofounders of the R language, Ross Ihaka, argued on the contrary that the future of statistical software is... Lisp: Back to the Future: Lisp as a Base for a Statistical Computing System. @Alex already pointed to the Clojure-based statistical environment Incanter, so maybe we will see a revival of Lisp-based software in the near future? :-)
First of all let me tell you that in my opinion the best tool of all by far is R, which has tons of libraries and utilities I am not going to enumerate here.
Let me expand the discussion about weka
There is a library for R, which is called RWeka, which you can easily install in R, and use many of the functionalities from this great program along with the ones in R, let me give you a code example for doing a simple decision tree read from a standard database that comes with this package (it is also very easy to draw the resulting tree but I am going to let you do the research about how to do it, which is in the RWeka documentation:
library(RWeka)
iris <- read.arff(system.file("arff", "iris.arff", package = "RWeka"))
classifier <- IBk(class ~., data = iris)
summary(classifier)
There are also several python libraries for doing this (python is very very easy to learn)
First let me enumerate the packages you can use, I am not going to go in detail about them; Weka (yes you have a library for python), NLKT (the most famous open source package for textmining besides datamining), statPy, sickits, and scipy.
There is also orange which is excellent (I will also talk about it later), here is a code example for doing a tree from the data in the table cmpart1, that also performs 10 folds validation, you can also graph the tree
import orange, orngMySQL, orngTree
data = orange.ExampleTable("c:\\python26\\orange\\cmpart1.tab")
domain=data.domain
n=10
buck=len(data)/n
l2=[]
for i in range(n):
tmp=[]
if i==n-1:
tmp=data[n*buck:]
else:
tmp=data[buck*i:buck*(i+1)]
l2.append(tmp)
train=[]
test=[]
di={'yy':0,'yn':0,'ny':0,'nn':0}
for i in range(n):
train=[]
test=[]
for j in range(n):
if j==i:
test=l2[i]
else:
train.extend(l2[j])
print "-----"
trai=orange.Example(domain, train)
tree = orngTree.TreeLearner(train)
for ins in test:
d1= ins.getclass()
d2=tree(ins)
print d1
print d2
ind=str(d1)+str(d2)
di[ind]=di[ind]+1
print di
To end with some other packages I used and found interesting
Orange:data visualization and analysis for novice and experts. Data mining through visual programming or Python scripting. Components for machine learning. Extensions for bioinformatics and text mining. (I personally recommend this, I used it a lot integrating it in python and it was excelent) I can send you some python code if you want me to.
ROSETTA: toolkit for analyzing tabular data within the framework of rough set theory. ROSETTA is designed to support the overall data mining and knowledge discovery process: From initial browsing and preprocessing of the data, via computation of minimal attribute sets and generation of if-then rules or descriptive patterns, to validation and analysis of the induced rules or patterns.(This I also enjoyed using very much)
KEEL:assess evolutionary algorithms for Data Mining problems including regression, classification, clustering, pattern mining and so on. It allows us to perform a complete analysis of any learning model in comparison to existing ones, including a statistical test module for comparison.
DataPlot: for scientific visualization, statistical analysis, and non-linear modeling. The target Dataplot user is the researcher and analyst engaged in the characterization, modeling, visualization, analysis, monitoring, and optimization of scientific and engineering processes.
Openstats: Includes A Statistics and Measurement Primer, Descriptive Statistics, Simple Comparisons, Analyses of Variance, Correlation, Multiple Regression, Interrupted Time Series, Multivariate Statistics, Non-Parametric Statistics, Measurement, Statistical Process Control, Financial Procedures, Neural Networks, Simulation.
Colin Gillespie mentioned BUGS, but a better option for Gibbs Sampling, etc, is JAGS.
If all you want to do is ARIMA, you can't beat X12-ARIMA, which is a gold-standard in the field and open source. It doesn't do real graphs (I use R to do that), but the diagnostics are a lesson on their own.
Venturing a bit farther afield to something I recently discovered and have just begun to learn...
ADMB (AD Model Builder), which does non-linear modeling based on the AUTODIF library, with MCMC and a few other features thrown in. It preprocesses and compiles the model down to a C++ executable and compiles it as a standalone app, which is supposed to be way faster than equivalent models implemented in R, MATLAB, etc. ADMB Project.
It started and is still most popular in the fisheries world, but looks quite interesting for other purposes. It does not have graphing or other features of R, and would most likely be used in conjunction with R.
If you want to work with Bayesian Networks in a GUI: SamIam is a nice tool. R has a couple of packages that also do this, but SamIam is very nice.
I really enjoy working with RooFit for easy proper fitting of signal and background distributions and TMVA for quick principal component analyses and modelling of multivariate problems with some standard tools (like genetic algorithms and neural networks, also does BDTs). They are both part of the ROOT C++ libraries which have a pretty heavy bias towards particle physics problems though.
I second that Jay. Why is R valuable? Here's a short list of reasons. http://www.inside-r.org/why-use-r. Also check out ggplot2 - a very nice graphics package for R. Some nice tutorials here.
This falls on the outer limits of 'statistical analysis', but Eureqa is a very user friendly program for data-mining nonlinear relationships in data via genetic programming. Eureqa is not as general purpose, but it does what it does fairly well, and the GUI is quite intuitive. It can also take advantage of the available computing power via the eureqa server.
Symbolic mathematics software can be a good support for statistics, too. Here are a few GPL ones I use from time to time:
All three are in active development.
Meta.Numerics is a .NET library with good support for statistical analysis.
Unlike R (an S clone) and Octave (a Matlab clone), it does not have a "front end". It is more like GSL, in that it is a library that you link to when you are writing your own application that needs to do statistical analysis. C# and Visual Basic are more common programming languages than C/C++ for line-of-business apps, and Meta.Numerics has more extensive support for statistical constructs and tests than GSL.