-1
$\begingroup$

I am trying to run a logistic regression in R on my data where my independent variables are 13 continuous variables and my dependent variable is binary. I want to segment my data so that I train on the first 80% and test on the last 20%. I have a total of 3750 rows of data so I utilize the first 3000 for training. I have written the following:

mydata<-totaldata[1:3000,2:15]
mylogit<-glm(mydata$TARGET ~ mydata$VAR1+mydata$VAR2+mydata$VAR3+mydata$VAR4+ #$
                             mydata$VAR5+mydata$VAR6+mydata$VAR7+mydata$VAR8+
                             mydata$VAR9+mydata$VAR10+mydata$VAR11+mydata$VAR12+
                             mydata$VAR13, family="binomial")

predictdata=totaldata[3001:3751,3:15]
in_frame<-data.frame(predictdata)
predictions=predict(mylogit,in_frame,type="response")

However I get the following warning message: Warning message: 'newdata' had 751 rows but variable(s) found have 3000 rows

Then when I look at predictions there are 3000 predictions not the 751 that I wanted. What can I do to fix this?

$\endgroup$
2
  • $\begingroup$ Do you only want to know how to get R to do this? If so, your question is off-topic for CV (see our FAQ), but on-topic for Stack Overflow. If you have a substantive question about the statistical aspects here, please edit your Q to clarify this; if not, flag it & we'll migrate it for you. (Please don't cross-post, though, SE explicitly discourages this.) $\endgroup$ – gung - Reinstate Monica Jan 9 '13 at 23:00
  • 2
    $\begingroup$ A couple of additional notes: You can just list your variables in the formula as VAR1+VAR2..., and then include a data=mydata argument. This approach might be easier for you. Also, the selected columns of mydata & predictdata differ (2:15, & 3:15). I suspect this is a typo. $\endgroup$ – gung - Reinstate Monica Jan 9 '13 at 23:04
1
$\begingroup$

Your problem arises from the fact you have specified the formula

mydata$TARGET ~ mydata$VAR1+mydata$VAR2+mydata$VAR3+mydata$VAR4+ #$
                         mydata$VAR5+mydata$VAR6+mydata$VAR7+mydata$VAR8+
                         mydata$VAR9+mydata$VAR10+mydata$VAR11+mydata$VAR12+
                         mydata$VAR13

This means, when you run predict, it is looking for variables with names like

`mydata$Var1`

This causes scoping issues as it will look for the column `mydata$Var1` within your newdata object (which don't exist), and eventually evaulate to mydata$Var1 (hence the warning as it has found the the object with length 3000 which conflicts with the size of newdata`.

In essence you have forced predict.glm to ignore the data in the newdata argument.

If you specify the formula in call to glm

mylogit<-glm(TARGET ~ VAR1 + VAR2 + VAR3 + VAR4 + VAR5 + VAR6 + VAR7 + VAR8 + 
   VAR9 + VAR10 + VAR11 + VAR12 + VAR13, family="binomial", data = mydata)

Then all should be well.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.