# Automating a search for correlation

Note: I am not a statistician, though I do have a mathematics background. I know what I would like to do, but I don't have deep subject knowledge. If you use any special jargon, please try to use standard forms I can search for online.

An Example: Assume I have a multivariate data set ($n \approx 500$ with ) wherein I would like to search for regression or correlation. I am hoping to find a list of possible predictors for one of the variables in my set. This is probably a very common task, so I hope there's a simple way to automate the process.

For example, suppose for each subject in my sample, they report the following five variables: Age, Weight, Income, Education Level, and Favorite Pokemon

I would like software to churn through this to find things like:

"People between $25$ and $35$ years of age, who make between $\$30{,}000$and$\$45{,}000$ per year, and hold Master's Degrees or higher, tend to prefer Water-Type Pokemon. This is true, independent of the person's Weight."

Or...

"Age and preference for Empoleon are negatively correlated, independent of education level, and only for people earning less than $\$40{,}000\$ per year."

Can I automate the process of looking for subsets and testing those for correlation to a given variable?

Follow-up Question: I have a data set, and one of the variables is strongly bi-modal. Would the procedure I suggest above help me find a correlate?

• Your description at "software to churn through this to find things" sounds like a decision tree or random forest. They are about as automatic as you can hope to get. – whuber Mar 9 '18 at 16:16

You can take a look at rule-based machine learning algorithms, which are designed to learn the general "rules" or correlations that you are talking about in a large data set of many features. Although, I think you will find this is going to be far from automatic and will require a fair amount of care and tuning.

Several issues arise.

First, the thing that you use as an example is not an example of correlation, but regression.

Second, if a variable is strongly bimodal, then the usual regression will make little sense.

Third, even if you limit it to OLS regression, the number of possible regressions is huge. You will get a lot of type I errors unless you use some huge correction factor; but if you use a huge correction, then the effect size (with an N like yours) will have to be so huge that anything found will be obvious.

Fourth, although I know I am fighting against a tide here, I don't like letting a computer do my thinking for me.

Finally, and this is not directed solely at you and not meant to be an attack, but why do people start off questions that require statistical expertise to answer with "I am not a statistician?" No one starts off "I am not a surgeon but I need to know how to (do some surgery)" nor "I am not an accountant but I need to do the taxes for a large corporation". I suggest hiring a statistician and a programmer, if you decide to go ahead with this.

• @ Peter Flom To your "Finally" point: Since we're being rhetorical, does a Pokemon example suggest I'm trying to do something extremely important like surgery or taxes for a large corporation? I can't answer for folks in general, but I put that in there so that (a) you might know that extremely technical answers will miss the mark, (b) I am open to the possibility that what I'm asking for won't really work and (c) (a hope, really) answers here might take teaching tone. Would you consider reframing your answer in light of this? – user138719 Mar 9 '18 at 13:46