Determining which variables affect which measurements

Question

What software (or method) can I use to determine correlations between any sets of variables and results?

Background: I would like to analyze sets of our archived data over several years. Knowledge of the product tested often leads to some obvious correlations (ie: fuel efficiency peaks at a particular speed). Many trends are not discovered until an investigation following some issue. I'm looking to find any trends that may not be as obvious when looking at the data.

Generally, I seem to find trends by making and testing a hypothesis about the data. However, the process of constructing a query, copying the results to Excel, and analyzing with Excel (sorting, formulas, statistics, graphing, etc) seems too tedious, inefficient, and slow.

I'm looking for software that would take a table of data, look for trends between all sets of variables (inputs) and attributes (outputs), and display the most likely correlations.

Answer 1

There are lots of statistical software packages out there that will speed up the process versus Excel. E.g., SAS, SPSS, Minitab, Stata, and R (the last of which is free and is favored by many people using this site). But don't be fooled into thinking you'll be able to click a few buttons and be done. From what you've said, you still have a lot of hard work in front of you, not least because some of the most valuable findings in your future may involve something other than linear correlation. Suppose two variables have an upside-down-U-shaped relationship. At low or high levels of X, Y will be low, but at some in-between level of X, Y will be maximized. To unearth such dynamics will require that you create many graphs such as scatterplots or that you perform tests such as ANOVAs with post hoc comparisons. Good luck in your explorations.