# How do you know which predictor value to use? Linear Regression

I'm trying to automate linear regression with R, although I don't really have a concrete background in statistics, I was wondering are there numerical techniques in determining wheither the predictor values is even worth trying to fit against a response value before attempting to do this in R:

lmfit <- lm(x ~ y)


My initial guess was covariance and correlation, but again what are the accepted values? Also what if* the relationship isn't linear.

In most scenarions (as mentioned by commenters already) one would base predictors based on an a priori hypothesis. However* in knowledge discovery, there are times where you don't know what the hypothesis is, hence my motivation towards formulating a set of rules whereby a quick numerical parameter would allow me to decide if a certain predictor variable should be added or removed in the linear regression model.

Note that the discussion above is not data dependent, it should work with any general data that may or may not have linear relationships between variables.

-
If your interest is in making substantive interpretations from your model, then what predictors "are worth trying" should be based on an a priori hypothesis you have about the data. –  Macro Jul 5 '12 at 16:55
If you're using the conventional notation of y as a response and x as a predictor, your R formula is then backwards. –  AdamO Jul 5 '12 at 17:20
I find this question difficult to understand. Can you say more about what you want to do? Eg, what are the date you're working with? What are you're goals? What is the larger situation? (etc.) You may find this helpful. –  gung Jul 5 '12 at 17:47
I've edited the description a bit more, hopefully that should help. –  chutsu Jul 5 '12 at 21:14

If you have a large number of variables, there is uncertainty regarding the choice of predictors. I think this is root of OP's problem. Bayesian model averaging (BMA) helps take this uncertainty into account.

Check out Adrian Raftery's page on BMA. Through BMA, you can select a subset of potentially helpful predictors. R packages exist for most of these methods.

http://www.stat.washington.edu/raftery/Research/bma.html

I found Raftery's paper on BMA modeling economic growth to be particularly interesting

http://www.stat.washington.edu/raftery/Research/PDF/Eicher2010.pdf

-

Most of our brush-ups with statistics are inference, they are casually (mis)reported in the news and come to influence people's decisions about diet, medical treatment options, college degrees, etc for weal or woe. Getting a true apples-to-apples comparison for the purposes of inference requires a great deal of thought about the nature of the association. The first step here is prespecifying your effect of interest. Letting numbers--and not science--guide these decisions is the surefire way to generate unreproducable (spurious) results. The p-value no longer means what you think it does, since other aspects of the data have guided what you have chosen to report as a result.

Suppose, for instance, I walked into a maternity ward and observed that, for whatever reason, there were 20 babies there and 18 were female. I would be shocked to see such a thing and say to myself, "The popular opinion is that the male to female ratio of humans is 1-1, but I now have evidence suggesting this belief may be incorrect, so I will formulate a testable hypothesis that the male-female ratio of newborns is in fact other than 1-1". I then analyze the data I used to generate the hypothesis, calculate a population proportion test and get $p=0.0008$. I conclude that there is definitive evidence suggesting that the ratio of male to female births I observed is inconsistent with the 1-1 ratio. I publish in JAMA and get a tenure position at a prestigious university. What do you suppose will happen when a competing researcher gathers data from a neighboring facility?

-
Could you explain how your answer has anything to do with the OPs question? –  Michael Chernick Jul 5 '12 at 17:50
Questions like these are often indicative of a lack of clear understanding for what purpose the data is collected. Using associations which are found in the data themselves to retrospectively generate the reason for which we might have collected the data generally leads to invalid results. This is certainly the case in confirmatory analyses but is also a big risk in exploratory analyses. I believe there's no "numerical way of determining if a predictor is even worth trying to fit in a model". –  AdamO Jul 5 '12 at 18:02
It seems to me that if you have a prediction problem and you are considering covariates to use for prediction that a variable that has no correlation (as shown by the data) with the respons ethat you intend to predict, then it is perfectly reasoanble to assume that that covariate is not worth conside3ring in the model. The OPs question was whether or not there are quick ways to notice this. It seems like a reasonable question. In general choosing covariates for a model can be difficult and the purpose of the data and the model do generally enter into what to choose. –  Michael Chernick Jul 5 '12 at 18:27
Nut there can be big clues as to what to rule out. –  Michael Chernick Jul 5 '12 at 18:27
yes but what does that have to do with ruling out obviously poor predictors?? –  Michael Chernick Jul 5 '12 at 19:10
@Macro Consider the example y=x$^3$ y is a nonlinear function of x that is monotonically increasing. Because it is nonlinear if we had data yi=xi$^3$ for i=1,2,...,n y and x would not have correlation 1 but it would be high. The same is true for yi=xi$^2$ for xi>0 but not for yi=xi$^2$ when the xi are uniformly distributed on [-1,1]. So my statement does not apply universally but my point really is that the function can be nonlinear without the correlation being low. –  Michael Chernick Jul 5 '12 at 19:20