Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote -10 down vote favorite

Twenty possible predictor variables in data set. One outcome variable.

Some of the predictor variables are not linear. So a standard linear multiple regression approach probably won't do. (And I do not want to transform the variables using some arbitrary "cubic," "quadratic" approach -- leave the underlying variables as they are.)

Example: Time of day. This oscillates regularly, over 24 hours. (Like a sine wave.)

There is considerable random chaos inherent in some variables.

Example: Highway traffic frequency.

Some curve fitting may be advisable to dampen or smooth out the chaos.

I'm looking for possible statistical packages (and specific procedures) that can accomplish the goal -- determining the two best predictor variables taken together of the twenty, and levels of each, that appear to best predict the outcome variable.

Considering all twenty variables, there are 190 possible combinations. Doing an analysis of each possible pair would take a great deal of effort. Best to find a statistical package that can run tests on each possible combination in one long go.

In summary: Software Package and Specific Procedure please. No need for a bunch of code. (If I want the code I'll ask in a follow-up.)

Thanks.

Nicholas Kormanik

share|improve this question
1  
Let's say you limit the possible relationships to linear, quadratic, and cubic ones. That increases the possibilities from 190 to 570. I don't know of a program that will test all such possibilities for you. I think you're going to have to exercise some human judgment. However you approach it, I hope you'll use some sort of crossvalidation. How big is your sample? – rolando2 May 16 '12 at 23:18
What do you mean by "levels of each" when talking about predictor variables? – Peter Ellis May 17 '12 at 1:59
What do you mean by "The predictor variables are not linear."? – Macro May 17 '12 at 3:12
1  
@nkormanik, why do you think "non-linear" predictors are a problem? In regression, you are conditioning on the predictor values - their marginal distribution only matters insofar as it impacts the standard errors. You don't need the predictors to be "linear" (using your definition of linear). – Macro May 17 '12 at 12:23
4  
"The predictor variables are not linear." doesn't make sense in isolation - in order to be "linear" they must be thought of as a linear function of some other variable which you said nothing about in the question. In any case whether the predictor is "linear" has nothing to do with whether the model relating it with a dependent variable should be linear or not. – Macro May 18 '12 at 0:43
show 5 more comments

1 Answer

up vote 6 down vote

Fitting your 190 models in one go is easy in software like R. See code below for an example. However, as @rolando2 suggests, some human judgement would go a long way. For example, for each of the 20 variables, could you look at them one at a time and work out the best way for them to be transformed or smoothed? (as you have started to do). Doing it bY brute force (eg fitting a cubic polynomial, as I do below) is not really recommended.

@rolando2's other essential point is that you should use some cross-validation. At a minimum, separate your data into a training and a testing set. There is lots of good advice on this elsewhere on the site.

Example of fitting 190 models, each with cubic relationships, below. This is meant to be more of an illustration as an answer than a recommendation. Basically, your conceptual and analytical tasks are much more complicated than mere software to fit lots of models.

SO, NOT REALLY RECOMMENDED AS AN APPROACH...

# set up data
expl <- as.data.frame(matrix(rnorm(200000), ncol=20))
names(expl) <- paste0("Var",1:20)
resp <- rnorm(10000)


pair <- combn(1:20,2)

# create somehwere to hold results

res <- list()

for (i in 1:190){
    x1 <- expl[ , pair[1,i]]
    x2 <- expl[ , pair[2,i]]
    res[[i]] <- lm(resp ~ poly(x1,3) + poly(x2,3))
}

names(res) <- paste0(names(expl)[pair[1,]], names(expl)[pair[2,]])
summary(res[[1]])$r.squared

# calculate the R squared for each
report <- unlist(lapply(res, function(x){summary(x)$r.squared}))

# print to screen, with highest R-squared last
report[order(report)]
share|improve this answer
1  
(+1) since this does precisely what the original poster asked for. One comment - if the effect of two predictors are being considered conjointly wouldn't you also want to include cross products $x_{1}x_{2}$, $x_{1}^2 x_{2}$, $x_{1}x_{2}^2$ and $x_{1}^2 x_{2}^2 $? Then this would be like fitting the 3rd order taylor series approximation of the regression function $f(x_1,x_2)$ – Macro May 17 '12 at 3:51
The example given at the top of the post posits two predictor variables -- time of day, highway traffic frequency. These two are then taken TOGETHER to predict some response variable. There seems no need for "linear, quadratic, cubic." Why complicate the matter?? Should be simple and straight-forward. Which two of the twenty possible predictor variables, and at what respective levels, used TOGETHER, do the best job in predicting the response? What statistical package seems best to use? What procedure in the package? Please try to explain so that others here can learn from your answer. – nkormanik May 17 '12 at 5:38
I presume you are not familiar with R. Perhaps you should look into some possible software solutions - there is plenty of information out there. My answer above is readily adaptable to the simpler situation you want, so I leave it to you to work out how to do that. – Peter Ellis May 17 '12 at 6:29
2  
As a tip for where the miscommunication is coming from - in your question you said a straightforward linear regression would not do, hence some of us tried to provide non-linear solutions which you seem to think is "complicating the matter". – Peter Ellis May 17 '12 at 6:32
4  
@nkormanik, re: "Why complicate the matter??" - if there are non-linear effects, including polynomial terms effectively gives a higher order approximation to the true regression function $f(x_1, x_2)$, similarly to how including more terms in a taylor series expansion improves the approximation. – Macro May 17 '12 at 12:18
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.