Significance of $r^2$ value

I know about $r^2$ tells you about the amount of variation that can be explained by the predictor variables. I have run a model in which the rsquare has value 0.3010 but has false positive rate of around 15.60%. So this model which is logit in nature predicts 84% cases right. I want to know two things:
1) Is this false positive rate significant, i.e. is this prediction good enough?
2) Does it matter if I $r^2$ is as low as 0.3 if my model is good in prediction as we know that $r^2$ is a value used for explaning and not predicting?

-
Does it predict new data with 85% accuracy, or part of the original data set that you held back for validation? –  richiemorrisroe Nov 5 '10 at 11:08
@richie..that rings an alarm..I didnt test it for any data. I ran a decision tree and it gave me say x false positives and y true positives than x/x+y is .156 which is misclassification..am i right then in saying that it has 84.6% accuracy ? –  ayush biyani Nov 5 '10 at 11:24
Which $R^2$ measure are your referring to? If I understand you question correctly, this is logistic regression, and you may find useful information on this related question, stats.stackexchange.com/questions/3559/…. –  chl Nov 5 '10 at 12:14
For future: when you don't know which tag to use, add for-retag. –  mbq Nov 5 '10 at 12:16
Isn't it $R^2$ rather than $r^2$ ? –  Dirk Eddelbuettel Nov 5 '10 at 13:41
add comment

1 Answer

First, no model is perfect unless it is over-fit. So, your false positive rate is not unusual. Is a false positive rate of 16% good or bad? If it is lower than the natural proportion in the data it is OK. If it is not it is really bad. The key is by how much your model reduces the error rate. That's measured in several ways as mentioned in the second paragraph.

Second, R Square is not a very well accepted measure of a model's robustness and explanatory power for Logit regression. I would not rely solely on it. There are other R Square measures earmarked for Logit regression. They include Pseudo R Square, Aldrich and Nelson R Square, and McFadden R Square. They all measure the reduction in error as captured by the -2LL calculation between the Baseline model that simply assumes that the probability for all observations is the average probability or the average proportion (if all values are 1 or 0). Another answer also mentions the Cox & Snell R Square and the Nagelkerke R Square. I suspect those are other variants on the same theme focusing on the model's error reduction.

I would re-frame the investigation on the robustness of your model by focusing on the mentioned different R Square measures. I would also calculate the Chi Square p value for the whole model. The difference in the -2LL between the baseline model and the model is a Chi Square value. The number of variables gives you the DF. The resulting Chi Square p value gives you a measure of statistical significance for the whole model (probability that results were due to randomness).

Maybe even more importantly, I would focus on the statistical significance of the independent variables you are using. The Wald statistics (really another Chi Square test) fits that purpose well. In this case, a variable regressed coefficient divided by the coefficient standard error gives you the Wald value. DF is always 1. And, the resulting p value conveys the statistical significance of this variable.

With the above, you can then remove or add variables and attempt to improve your model.

-
how do you calculate the Chi square p value for the whole model ? Forgive my ignorance. –  ayush biyani Nov 7 '10 at 15:25
@ayush. I describe the calculation in the third paragraph. In terms of actually calculating the resulting Chi Square p value, you can use an Excel formula. If I remember correctly it is either CHIDIST or CHITEST. –  Gaetan Lion Nov 7 '10 at 18:59
@gaeton..Thanks. –  ayush biyani Nov 9 '10 at 14:42
add comment