Software for easy-yet-robust data exploration

Question

In my attempts to fight spreadsheet mayhem, I am often evangelical in pushing for more robust tools such as true statistics software (R, Stata, and the like). Recently, I was challenged on this view by someone who stated flat out that they simply will not learn to program. I would like to provide them with data analysis tools that require no programming (but ideally which would extend to programming if they decide to dip a toe into the water later). What packages are out there for data exploration that I can recommend with a straight face?

Answer 1

I program in Python for 95% of my work and the rest in R or MATLAB or IDL/PV-WAVE (and soon SAS). But, I am in an environment where time-to-results is often a huge driver of the analysis chosen and so I often use point-and-click tools as well. In my experience, there is no single, robust, flexible GUI tool for doing analytics, much like there is not a single language. I typically cobble together a collection of the following free and commercial software

• Weka
• KNIME
• Excel and its plugins (like Solver)
• Alteryx
• MVP Stats

I have not used JMP, Stata, Statistica, etc, but would like to.

Using these tools involves learning different GUIs and multiple abstractions of modeling, which is a pain at the time but let's me get faster ad hoc results later. I am in the same boat as the OP because while most of the folks I work with are really smart, they do not care to learn a language, nor multiple GUIs and application specific terminology. So, I have resigned myself to accepting that Excel drives 90% of analysis in the business world. Accordingly, I am looking in to using things like pyinex to let me provide better analytics to the same Excel presentation layer that the vast majority of my colleagues expect.

UPDATE: Continuing down the Do-modeling-with-programming-but-make-Excel-the-presentation-layer theme, I just came across this guy's website offering Tufte-style graphics to embed in Excel cells. Simply awesome and free!

Answer 2

Some people think of programming as simply entering a command line statement. At that point then perhaps you are a bit lost in encouraging them. However, if they are using spreadsheets already then they already have to enter formulas. These are akin to command line statements. If they really mean they don't want to do any programming in the sense of logical and automated analysis then you can tell them that they can still do the analyses in R or Stata without any programming at all.

If they can do their stats in the spreadsheet... all that they want to do... then all of the statistical analyses they wish to accomplish can be done without 'programming' in R or Stata as well. They could arrange and organize the data in the spreadsheet and then just export it as text. Then the analysis is carried out without any programming at all.

That's how I do intro to R sometimes. No programming is required to do the data analysis you could do in a spreadsheet.

If you get them hooked that way then just reel the fish in slowly... :) In a couple of years compliment them on what a good programmer they've become.

You might also want to show this document to your colleagues or at least read it yourself to better make your points.

Answer 3

I'm going to put a pitch in here for JMP. I have a couple reasons why it's my preferred non-programming data exploration tool of choice:

1. Really good visualization tools. More most basic EDA-type plots, it's as good as R is, and considerably easier to use for producing something approaching a publication-ready plot. It's also got some extremely flexible visualization tools, so you can twist and bend your data around to get the full story.
2. Surprisingly powerful. It took me until my...4th year of grad school to find something JMP couldn't do right out of the box. That's not bad.
3. Scriptability. This is a big thing for me. The main weakness of GUIs is that its very hard to replicate what you did. JMP allows you to script the GUI - and generating those scripts is pretty point and click.

Answer 4

As far as exploratory (possibly interactive) data analysis is concerned, I would suggest to take a look at:

• Weka, originally targets data-mining applications, but can be used for data summaries.
• Mondrian, for interactive data visualization.
• KNIME, which relies on the idea of building data flows and is compatible with Weka and R.

All three accept data in arff or csv format.

In my view, Stata does not require so much programming expertise. This is even part of its attractiveness, in fact: Most of basic analysis can be done by point-and-click user actions, with dialog boxes for customizing specific parameters, say, for prediction in a linear model. The same applies, albeit to a lesser extent, to R when you use external GUIs like Rcmdr, Deducer, etc. as said by @gsk3.

Answer 5

I can recommend Tableau as a good tool for data exploration and visualization, simply because of the different ways that you can explore and view the data, simply by dragging and dropping. The graphs are fairly sharp and you can easily output to PDF for presentation purposes. If you want you can extend it with some "programming". I regularly use this tool along with "R" and SAS and they all work together well.

Answer 6

As John said, data exploration doesn't require much programming in R. Here's a list of data exploration commands you can give people. (I just came up with this; you can surely expand it.)

Export the data from whatever package it's in. (Exporting numerical data without quotation marks is convenient.) Then read the data in R.

ChickWeight=read.csv('chickweight.csv')

Make a table.

table(ChickWeight$Diet)

Let R guess what sort of graphic to give you. It sometimes works very nicely.

plot(ChickWeight)
plot(ChickWeight$weight)
plot(ChickWeight$weight~ChickWeight$Diet)

A bunch of specific plotting functions work quite simply on single variables.

hist(ChickWeight$weight)

Taking subsets

plot(subset(ChickWeight,Diet=='2'))

SQL-like syntax in case people are used to that (more here)

library(sqldf)
plot(sqldf('select * from ChickWeight where Diet == "2"'))

PCA (You'd have more than two variables of course.)

princomp(~ ChickWeight$weight + ChickWeight$Time)

Answer 7

This is more of a lament than an answer...

The best software I've seen for this is Arc, which is built on top of Xlisp-Stat. It's fantastic software for data exploration with lots of built in interactive graphics, as well as lots of statistical inference capabilities. In my opinion nothing else has come close to its ease of use for data exploration and ability to extend it further with Lisp programming. In my opinion, interactivity in R is just starting to be able to used in ways like Arc, ten long years later. And as far as I know, no one has yet used these capabilities to build up an interactive interface that is anywhere near as useful as Arc.

Unfortunately, it never really caught on so the developers have since almost all switched to working in R; it was last updated in July of 2004. The PC and Linux/Unix versions still work and might be worth a try, depending on your needs; for Macs the best option is to try the Linux/Unix version under X11, I've gotten it working on a couple systems that way. The Mac version mentioned on the site only works on "Classic" Macs.

I'll also mention briefly Mondrian, which I've only tried briefly, but seems to have terrific graphical interactivity for data exploration, though (as I recall) no easy way to extend the abilities or do statistical inference.

Answer 8

A new software system that looks promising for this purpose is Deducer, built on top of R. Unfortunately, being new, I suspect it does not yet cover the breadth of questions that people might ask, but it does meet the toe-in-the-water criterion of leading people towards a true package should they so decide later.

I've also used JMP in the past, which had a nice interactivity to it. I am worried that some of the interface might be too complicated for these purposes. And it's non-free, which makes it harder for potential spreadsheet refugees to try out on a whim.

---

There's also Rattle which looks somewhat promising.

Answer 9

Anyone who answers R, or any of it's "GUIs" didn't read the question.

There is a program specifically designed for this and it's called JMP. Yes, it's expensive, though it has a free trial, and is incredibly cheap for students or college staff (like $50 cheap).

There is also RapidMiner, which is a workflow-based GUI for data mining and statistical analysis. It's free and open source.

Answer 10

Well, this particular tool is popular in my industry (though it is not industry-specific by design): http://www.umetrics.com/simca

It allows you to do latent variable type multivariate analysis (PCA and PLS), and it includes all the attendant interpretative plots / calculations and interrogation tools like contribution plots, variable importance plots, Q2 calculations etc.

It is often used on high-dimensional (and often highly correlated/collinear) industrial datasets where OLS/MLR type methods are unsuitable (e.g. info from a boatload of sensors, log info, etc.).

It operates in a fully GUI environment, and the user does not have to write a single line of code. Unfortunately it is not free, and cannot be extended via programming.

Answer 11

In my opinion, if you don't code yourself the test, you are prone to errors and misunderstandings of the results.

I think that you should recommend them to hire a statistician that has computer skills.

If it is to do always the same thing, then indeed you can use a small tool (blackbox) that will do the stuff. But I am not sure this is still called data exploration.

Answer 12

I would recommend John Fox's R package called R commander:

http://socserv.mcmaster.ca/jfox/Misc/Rcmdr/

It creates a user interface similar to SPSS (or the like) that is great for beginners and does not require the user to input any code at all. It is all done via drop-down boxes (you can even minimize the R console while working).

To me, the benefit of this package is that you can take advantage of all the great computational ability of R while having a user interface that is completely operational for beginners.

Answer 13

For the exploration of what data contain and cleaning it up Google Refine is a pretty good GUI. It's much more powerful for the preparation and cleaning than something like Excel. Then switch to something like R-Commander for your analyses.