In my attempts to fight spreadsheet mayhem, I am often evangelical in pushing for more robust tools such as true statistics software (R, Stata, and the like). Recently, I was challenged on this view by someone who stated flat out that they simply will not learn to program. I would like to provide them with data analysis tools that require no programming (but ideally which would extend to programming if they decide to dip a toe into the water later). What packages are out there for data exploration that I can recommend with a straight face?
$\begingroup$
$\endgroup$
8
asked Apr 2, 2011 at 13:32
-
5$\begingroup$ @gsk3 - interesting question. I would have a guess that this "programaphobe" does not have to deal with the complicated spreadsheets? perhaps he/she would like to live up to their ideology and volunteer to do the hard work with the spreadsheets themselves? :) $\endgroup$– probabilityislogicApr 2, 2011 at 14:37
-
$\begingroup$ @probabilityislogic: The "programaphobe" does deal with them once they cause problems, but has a whole bag of tricks for dealing with the results. I've tried the classic anti-spreadsheet arguments to little avail (well, some avail, since they're willing to consider this style of solution!). $\endgroup$– Ari B. FriedmanApr 2, 2011 at 14:46
-
$\begingroup$ @gsk3 - ah ha, so its not the program that is the issue per se, but this person will no longer be "the expert" and will have to start from the bottom? $\endgroup$– probabilityislogicApr 2, 2011 at 16:51
-
2$\begingroup$ @probabilityislog - I think sunk costs are a large part of the issue, yes. Some of those costs are doubtless social, and some of them are time and efficiency losses while they learn the new way of doing things. $\endgroup$– Ari B. FriedmanApr 2, 2011 at 17:53
-
3$\begingroup$ I think that this comic might be appropriate representation of the situation. If the person you are refering is dead set against programming, this may be too strong a preference to go against. A better strategy might be to point out the deficiencies of spreadsheet based analysis. Try for example in new Excel file to enter values $1\frac{1}{6}$ and $1\frac{1}{4}$ and sum them (this works on Excel 2007). Then explain that no self-respecting statistical package will provide similar mistake without any warning and work from that. $\endgroup$– mpiktasApr 4, 2011 at 10:53
|
Show 3 more comments
14 Answers
$\begingroup$
$\endgroup$
1
I program in Python for 95% of my work and the rest in R or MATLAB or IDL/PV-WAVE (and soon SAS). But, I am in an environment where time-to-results is often a huge driver of the analysis chosen and so I often use point-and-click tools as well. In my experience, there is no single, robust, flexible GUI tool for doing analytics, much like there is not a single language. I typically cobble together a collection of the following free and commercial software
I have not used JMP, Stata, Statistica, etc, but would like to.
Using these tools involves learning different GUIs and multiple abstractions of modeling, which is a pain at the time but let's me get faster ad hoc results later. I am in the same boat as the OP because while most of the folks I work with are really smart, they do not care to learn a language, nor multiple GUIs and application specific terminology. So, I have resigned myself to accepting that Excel drives 90% of analysis in the business world. Accordingly, I am looking in to using things like pyinex to let me provide better analytics to the same Excel presentation layer that the vast majority of my colleagues expect.
UPDATE: Continuing down the Do-modeling-with-programming-but-make-Excel-the-presentation-layer theme, I just came across this guy's website offering Tufte-style graphics to embed in Excel cells. Simply awesome and free!
answered Apr 4, 2011 at 2:11
-
1$\begingroup$ I'll say that JMP is quite good. Even if one is very proficient in R, there are times when JMP is a faster way to navigate and analyze data. $\endgroup$– IteratorFeb 19, 2012 at 12:15
$\begingroup$
$\endgroup$
1
As far as exploratory (possibly interactive) data analysis is concerned, I would suggest to take a look at:
- Weka, originally targets data-mining applications, but can be used for data summaries.
- Mondrian, for interactive data visualization.
- KNIME, which relies on the idea of building data flows and is compatible with Weka and R.
All three accept data in arff or csv format.
In my view, Stata does not require so much programming expertise. This is even part of its attractiveness, in fact: Most of basic analysis can be done by point-and-click user actions, with dialog boxes for customizing specific parameters, say, for prediction in a linear model. The same applies, albeit to a lesser extent, to R when you use external GUIs like Rcmdr, Deducer, etc. as said by @gsk3.
-
$\begingroup$ +1 for Stata. You can do all of the basics through point-n-click, but it also spits out the command that point-n-click generated so you can learn/modify it. It's also spreadsheet-like in how you interface with its data, though you don't put formulas in cells of course. $\endgroup$– WayneNov 4, 2014 at 18:00
$\begingroup$
$\endgroup$
4
Some people think of programming as simply entering a command line statement. At that point then perhaps you are a bit lost in encouraging them. However, if they are using spreadsheets already then they already have to enter formulas. These are akin to command line statements. If they really mean they don't want to do any programming in the sense of logical and automated analysis then you can tell them that they can still do the analyses in R or Stata without any programming at all.
If they can do their stats in the spreadsheet... all that they want to do... then all of the statistical analyses they wish to accomplish can be done without 'programming' in R or Stata as well. They could arrange and organize the data in the spreadsheet and then just export it as text. Then the analysis is carried out without any programming at all.
That's how I do intro to R sometimes. No programming is required to do the data analysis you could do in a spreadsheet.
If you get them hooked that way then just reel the fish in slowly... :) In a couple of years compliment them on what a good programmer they've become.
You might also want to show this document to your colleagues or at least read it yourself to better make your points.
-
1$\begingroup$ Good points, but there some caveats here: Excel has tool tips that pop up when entering a formula in a cell and so it makes it much easier to "program" simple tasks. Also, chances are the person next to you can help if you get stuck since there are lots of Excel power users. And, Excel comes installed. Convincing someone to try R involves helping them set it up and learn how to leverage open source software (and which forums to search, which of the umpteen bootstrap packages should be used, etc). The programming is in some sense the easy part. Its the ecosystem that is the hard part. $\endgroup$ Apr 4, 2011 at 2:18
-
$\begingroup$ Also there is no real formatting issues in Excel... In R sometimes one package uses xts, another one data.frames... it's a mess for a newbie, that's true $\endgroup$ Apr 4, 2011 at 6:18
-
$\begingroup$ fRed, I agree that there can be issues in data formatting but keep in mind that my response was qualified as analysis that already could be done in a spreadsheet. That's not a lot of varied analysis and would generally be under one data format. $\endgroup$– JohnApr 4, 2011 at 16:55
-
$\begingroup$ There is somewhat more overhead in "programming" with R than with Excel, and these are the things that drive users crazy. If R had a top notch GUI akin to a Pivot Table, maybe. But until then... $\endgroup$ Apr 4, 2011 at 17:23
$\begingroup$
$\endgroup$
2
I'm going to put a pitch in here for JMP. I have a couple reasons why it's my preferred non-programming data exploration tool of choice:
- Really good visualization tools. More most basic EDA-type plots, it's as good as R is, and considerably easier to use for producing something approaching a publication-ready plot. It's also got some extremely flexible visualization tools, so you can twist and bend your data around to get the full story.
- Surprisingly powerful. It took me until my...4th year of grad school to find something JMP couldn't do right out of the box. That's not bad.
- Scriptability. This is a big thing for me. The main weakness of GUIs is that its very hard to replicate what you did. JMP allows you to script the GUI - and generating those scripts is pretty point and click.
-
$\begingroup$ +1 for JMP. It's the best basic stats gui I've ever used. $\endgroup$– ZachNov 6, 2011 at 16:22
-
1$\begingroup$ Agreed (well with point #1). In some ways, it is even better than R for the EDA stuff, and can be inserted nicely into an iterative workflow. If one is already working without R, then JMP is a reasonable way to work without adding R. Because it can also link to R, an R programmer can create tools that then plug in to JMP, allowing the Eloi, errr, user to continue to believe that everything is just dandy. $\endgroup$– IteratorFeb 19, 2012 at 12:18
$\begingroup$
$\endgroup$
I can recommend Tableau as a good tool for data exploration and visualization, simply because of the different ways that you can explore and view the data, simply by dragging and dropping. The graphs are fairly sharp and you can easily output to PDF for presentation purposes. If you want you can extend it with some "programming". I regularly use this tool along with "R" and SAS and they all work together well.
$\begingroup$
$\endgroup$
As John said, data exploration doesn't require much programming in R. Here's a list of data exploration commands you can give people. (I just came up with this; you can surely expand it.)
Export the data from whatever package it's in. (Exporting numerical data without quotation marks is convenient.) Then read the data in R.
ChickWeight=read.csv('chickweight.csv')
Make a table.
table(ChickWeight$Diet)
Let R guess what sort of graphic to give you. It sometimes works very nicely.
plot(ChickWeight)
plot(ChickWeight$weight)
plot(ChickWeight$weight~ChickWeight$Diet)
A bunch of specific plotting functions work quite simply on single variables.
hist(ChickWeight$weight)
Taking subsets
plot(subset(ChickWeight,Diet=='2'))
SQL-like syntax in case people are used to that (more here)
library(sqldf)
plot(sqldf('select * from ChickWeight where Diet == "2"'))
PCA (You'd have more than two variables of course.)
princomp(~ ChickWeight$weight + ChickWeight$Time)
answered Apr 3, 2011 at 14:49
$\begingroup$
$\endgroup$
2
This is more of a lament than an answer...
The best software I've seen for this is Arc, which is built on top of Xlisp-Stat. It's fantastic software for data exploration with lots of built in interactive graphics, as well as lots of statistical inference capabilities. In my opinion nothing else has come close to its ease of use for data exploration and ability to extend it further with Lisp programming. In my opinion, interactivity in R is just starting to be able to used in ways like Arc, ten long years later. And as far as I know, no one has yet used these capabilities to build up an interactive interface that is anywhere near as useful as Arc.
Unfortunately, it never really caught on so the developers have since almost all switched to working in R; it was last updated in July of 2004. The PC and Linux/Unix versions still work and might be worth a try, depending on your needs; for Macs the best option is to try the Linux/Unix version under X11, I've gotten it working on a couple systems that way. The Mac version mentioned on the site only works on "Classic" Macs.
I'll also mention briefly Mondrian, which I've only tried briefly, but seems to have terrific graphical interactivity for data exploration, though (as I recall) no easy way to extend the abilities or do statistical inference.
answered Nov 5, 2011 at 19:30
-
$\begingroup$ Hadn't heard about Arc but going to check it out. Thanks. $\endgroup$ Nov 5, 2011 at 20:04
-
1$\begingroup$ (+1) Good to hear from the Lisp world again. I'm too a fan of xlispstat (and appreciate that Luke Tierney is being so active in the R project). $\endgroup$– chlNov 5, 2011 at 20:46
$\begingroup$
$\endgroup$
1
A new software system that looks promising for this purpose is Deducer, built on top of R. Unfortunately, being new, I suspect it does not yet cover the breadth of questions that people might ask, but it does meet the toe-in-the-water criterion of leading people towards a true package should they so decide later.
I've also used JMP in the past, which had a nice interactivity to it. I am worried that some of the interface might be too complicated for these purposes. And it's non-free, which makes it harder for potential spreadsheet refugees to try out on a whim.
There's also Rattle which looks somewhat promising.
answered Apr 2, 2011 at 13:35
-
$\begingroup$ Regarding JMP - If I'm not mistaken, there are trial versions and academic licenses that are within the reasonable. $\endgroup$– IteratorFeb 19, 2012 at 12:19
$\begingroup$
$\endgroup$
For the exploration of what data contain and cleaning it up the former Google Refine, now Open Refine, is a pretty good GUI. It's much more powerful for the preparation and cleaning than something like Excel. Then switch to something like R-Commander for your analyses.
$\begingroup$
$\endgroup$
7
Anyone who answers R, or any of it's "GUIs" didn't read the question.
There is a program specifically designed for this and it's called JMP. Yes, it's expensive, though it has a free trial, and is incredibly cheap for students or college staff (like $50 cheap).
There is also RapidMiner, which is a workflow-based GUI for data mining and statistical analysis. It's free and open source.
answered Apr 2, 2011 at 22:53
-
1$\begingroup$ @Neil Why is it so? The OP specified "but ideally which would extend to programming...". R has been plugged in almost all commercial (e.g., SPSS, JMP, Statistica) or free (Knime, Rapidminer) statistical software and Rserve can be used as a backend to communicate with R (e.g., with the plink software for genetic studies), and it is used for this purpose. The Free Trial version of JMP doesn't let you work with external datasets, am I wrong? (Interestingly, all of us who answered R, and an alternative software, were downvoted.) $\endgroup$– chlApr 3, 2011 at 9:02
-
$\begingroup$ i like R and am in favour of it, but people that want to learn stats/datamining and don't know how to program really can't use it. and i've played with all of its guis and they are just horrible compared to jmp. i am just waiting for the day when R gets a gui like jmp, then all will be well w the universe. $\endgroup$ Apr 3, 2011 at 16:52
-
$\begingroup$ -1 for suggestion that people do not read questions before answering and for not providing any real arguments why R is worse than JMP or RapidMiner. Note that in general I agree that R is not an answer to everything, but this topic is a bit contraversial with the potential to get really nasty. Hence the downvote. $\endgroup$– mpiktasApr 4, 2011 at 10:42
-
$\begingroup$ R is great, but it is not easy. That's what he asked for in the title. $\endgroup$ Apr 4, 2011 at 20:01
-
$\begingroup$ @Neil, yes, but you can argue that there is no such thing as easy-yet-robust data exploration. $\endgroup$– mpiktasApr 5, 2011 at 13:03
$\begingroup$
$\endgroup$
Well, this particular tool is popular in my industry (though it is not industry-specific by design): http://www.umetrics.com/simca
It allows you to do latent variable type multivariate analysis (PCA and PLS), and it includes all the attendant interpretative plots / calculations and interrogation tools like contribution plots, variable importance plots, Q2 calculations etc.
It is often used on high-dimensional (and often highly correlated/collinear) industrial datasets where OLS/MLR type methods are unsuitable (e.g. info from a boatload of sensors, log info, etc.).
It operates in a fully GUI environment, and the user does not have to write a single line of code. Unfortunately it is not free, and cannot be extended via programming.
$\begingroup$
$\endgroup$
1
In my opinion, if you don't code yourself the test, you are prone to errors and misunderstandings of the results.
I think that you should recommend them to hire a statistician that has computer skills.
If it is to do always the same thing, then indeed you can use a small tool (blackbox) that will do the stuff. But I am not sure this is still called data exploration.
answered Apr 4, 2011 at 4:11
-
1$\begingroup$ I more or less agree with the sentiment, but I don't think this high-minded view is really possible in many settings. $\endgroup$ Apr 4, 2011 at 5:19
$\begingroup$
$\endgroup$
I would recommend John Fox's R package called R commander:
http://socserv.mcmaster.ca/jfox/Misc/Rcmdr/
It creates a user interface similar to SPSS (or the like) that is great for beginners and does not require the user to input any code at all. It is all done via drop-down boxes (you can even minimize the R console while working).
To me, the benefit of this package is that you can take advantage of all the great computational ability of R while having a user interface that is completely operational for beginners.
$\begingroup$
$\endgroup$
Another useful tool, although just for Windows, is Spotfire -- I found it quite useful for quickly looking at various histograms and scatter plots for single and pairs of variables. A research tool that helps you rank single variables as well as pairs based on simple statistics -- Hierarchical Clustering Explorer from HCIL. It is nice for finding most interesting variables/pairs of variables.
