# How do you know which predictor value to use?

I'm trying to automate linear regression with R, although I don't really have a concrete background in statistics. I was wondering:

Are there numerical techniques in determining whether the predictor values is even worth trying to fit against a response value before attempting to do this in R:

lmfit <- lm(x ~ y)


My initial guess was covariance and correlation, but again what are the accepted values? Also, what if the relationship between response and predictor isn't linear.

In most scenarios (as mentioned by commenters already) one would base predictors based on an a priori hypothesis. However; in knowledge discovery, there are times where you don't know what the hypothesis is, hence my motivation towards formulating a set of rules whereby a quick numerical parameter would allow me to decide if a certain predictor variable should be added or removed in the linear regression model.

Note that the discussion above is not data dependent, it should work with any general data that may or may not have linear relationships between variables.

• If your interest is in making substantive interpretations from your model, then what predictors "are worth trying" should be based on an a priori hypothesis you have about the data. – Macro Jul 5 '12 at 16:55
• If you're using the conventional notation of y as a response and x as a predictor, your R formula is then backwards. – AdamO Jul 5 '12 at 17:20
• I find this question difficult to understand. Can you say more about what you want to do? Eg, what are the date you're working with? What are you're goals? What is the larger situation? (etc.) You may find this helpful. – gung - Reinstate Monica Jul 5 '12 at 17:47
• I've edited the description a bit more, hopefully that should help. – chutsu Jul 5 '12 at 21:14

Most of our brush-ups with statistics are inference, they are casually (mis)reported in the news and come to influence people's decisions about diet, medical treatment options, college degrees, etc for weal or woe. Getting a true apples-to-apples comparison for the purposes of inference requires a great deal of thought about the nature of the association. The first step here is prespecifying your effect of interest. Letting numbers--and not science--guide these decisions is the surefire way to generate unreproducable (spurious) results. The p-value no longer means what you think it does, since other aspects of the data have guided what you have chosen to report as a result.

Suppose, for instance, I walked into a maternity ward and observed that, for whatever reason, there were 20 babies there and 18 were female. I would be shocked to see such a thing and say to myself, "The popular opinion is that the male to female ratio of humans is 1-1, but I now have evidence suggesting this belief may be incorrect, so I will formulate a testable hypothesis that the male-female ratio of newborns is in fact other than 1-1". I then analyze the data I used to generate the hypothesis, calculate a population proportion test and get $p=0.0008$. I conclude that there is definitive evidence suggesting that the ratio of male to female births I observed is inconsistent with the 1-1 ratio. I publish in JAMA and get a tenure position at a prestigious university. What do you suppose will happen when a competing researcher gathers data from a neighboring facility?

• Could you explain how your answer has anything to do with the OPs question? – Michael R. Chernick Jul 5 '12 at 17:50
• Questions like these are often indicative of a lack of clear understanding for what purpose the data is collected. Using associations which are found in the data themselves to retrospectively generate the reason for which we might have collected the data generally leads to invalid results. This is certainly the case in confirmatory analyses but is also a big risk in exploratory analyses. I believe there's no "numerical way of determining if a predictor is even worth trying to fit in a model". – AdamO Jul 5 '12 at 18:02
• It seems to me that if you have a prediction problem and you are considering covariates to use for prediction that a variable that has no correlation (as shown by the data) with the respons ethat you intend to predict, then it is perfectly reasoanble to assume that that covariate is not worth conside3ring in the model. The OPs question was whether or not there are quick ways to notice this. It seems like a reasonable question. In general choosing covariates for a model can be difficult and the purpose of the data and the model do generally enter into what to choose. – Michael R. Chernick Jul 5 '12 at 18:27
• Nut there can be big clues as to what to rule out. – Michael R. Chernick Jul 5 '12 at 18:27
• yes but what does that have to do with ruling out obviously poor predictors?? – Michael R. Chernick Jul 5 '12 at 19:10

You do not want to automate linear regression. The choice of variables to consider in a model is difficult and requires serious thought. However as far as quickly eliminating variables is concerned I do think that if the correlation between the covariate and the response is very low then you can rule it out. Linearity of the relationship is not an issue. If there is a strong relationship that is not linear you should still see significant correlation and you can always look at nonparametric correlations like Spearman's and Kendall's to identify monotonic or other nonlinear relationships.

• Michael, can you clarify "If there is a strong relationship that is not linear you should still see significant correlation" because, taken literally, you know that's not true. – Macro Jul 5 '12 at 18:52
• @Macro Consider the example y=x$^3$ y is a nonlinear function of x that is monotonically increasing. Because it is nonlinear if we had data yi=xi$^3$ for i=1,2,...,n y and x would not have correlation 1 but it would be high. The same is true for yi=xi$^2$ for xi>0 but not for yi=xi$^2$ when the xi are uniformly distributed on [-1,1]. So my statement does not apply universally but my point really is that the function can be nonlinear without the correlation being low. – Michael R. Chernick Jul 5 '12 at 19:20
• I really only meant to say that often a strong relationship particularly a monotne relationship will have high correlation even though the relationship is nonlinear. – Michael R. Chernick Jul 5 '12 at 19:20
• Once you've added the word "monotone", we're in agreement. – Macro Jul 5 '12 at 19:20
• @chutsu Advising against something is different from saying that it is impossible. Given a set of potential predictor variables you can certainly automate a procedure for ruling out variables that don't correlate at all with the response. Stepwise selection methods are examples of automated procedures to start with a collection of variables and systematically add and subtract from the list until a final set is selected. This does not include interaction terms but you could automate the introduction of interaction terms also. It is possible. But the connotation is mindless manipulation. – Michael R. Chernick Jul 5 '12 at 21:34

If you have a large number of variables, there is uncertainty regarding the choice of predictors. I think this is root of OP's problem. Bayesian model averaging (BMA) helps take this uncertainty into account.

Check out Adrian Raftery's page on BMA. Through BMA, you can select a subset of potentially helpful predictors. R packages exist for most of these methods.

http://www.stat.washington.edu/raftery/Research/bma.html

I found Raftery's paper on BMA modeling economic growth to be particularly interesting

http://www.stat.washington.edu/raftery/Research/PDF/Eicher2010.pdf