3
$\begingroup$

I know about $r^2$ tells you about the amount of variation that can be explained by the predictor variables. I have run a model in which the rsquare has value 0.3010 but has false positive rate of around 15.60%. So this model which is logit in nature predicts 84% cases right. I want to know two things:
1) Is this false positive rate significant, i.e. is this prediction good enough?
2) Does it matter if I $r^2$ is as low as 0.3 if my model is good in prediction as we know that $r^2$ is a value used for explaning and not predicting?

$\endgroup$
5
  • 1
    $\begingroup$ Does it predict new data with 85% accuracy, or part of the original data set that you held back for validation? $\endgroup$ Nov 5, 2010 at 11:08
  • $\begingroup$ @richie..that rings an alarm..I didnt test it for any data. I ran a decision tree and it gave me say x false positives and y true positives than x/x+y is .156 which is misclassification..am i right then in saying that it has 84.6% accuracy ? $\endgroup$ Nov 5, 2010 at 11:24
  • 1
    $\begingroup$ Which $R^2$ measure are your referring to? If I understand you question correctly, this is logistic regression, and you may find useful information on this related question, stats.stackexchange.com/questions/3559/…. $\endgroup$
    – chl
    Nov 5, 2010 at 12:14
  • 1
    $\begingroup$ For future: when you don't know which tag to use, add for-retag. $\endgroup$
    – user88
    Nov 5, 2010 at 12:16
  • 4
    $\begingroup$ Isn't it $R^2$ rather than $r^2$ ? $\endgroup$ Nov 5, 2010 at 13:41

1 Answer 1

4
$\begingroup$

First, no model is perfect unless it is over-fit. So, your false positive rate is not unusual. Is a false positive rate of 16% good or bad? If it is lower than the natural proportion in the data it is OK. If it is not it is really bad. The key is by how much your model reduces the error rate. That's measured in several ways as mentioned in the second paragraph.

Second, R Square is not a very well accepted measure of a model's robustness and explanatory power for Logit regression. I would not rely solely on it. There are other R Square measures earmarked for Logit regression. They include Pseudo R Square, Aldrich and Nelson R Square, and McFadden R Square. They all measure the reduction in error as captured by the -2LL calculation between the Baseline model that simply assumes that the probability for all observations is the average probability or the average proportion (if all values are 1 or 0). Another answer also mentions the Cox & Snell R Square and the Nagelkerke R Square. I suspect those are other variants on the same theme focusing on the model's error reduction.

I would re-frame the investigation on the robustness of your model by focusing on the mentioned different R Square measures. I would also calculate the Chi Square p value for the whole model. The difference in the -2LL between the baseline model and the model is a Chi Square value. The number of variables gives you the DF. The resulting Chi Square p value gives you a measure of statistical significance for the whole model (probability that results were due to randomness).

Maybe even more importantly, I would focus on the statistical significance of the independent variables you are using. The Wald statistics (really another Chi Square test) fits that purpose well. In this case, a variable regressed coefficient divided by the coefficient standard error gives you the Wald value. DF is always 1. And, the resulting p value conveys the statistical significance of this variable.

With the above, you can then remove or add variables and attempt to improve your model.

$\endgroup$
2
  • $\begingroup$ how do you calculate the Chi square p value for the whole model ? Forgive my ignorance. $\endgroup$ Nov 7, 2010 at 15:25
  • 1
    $\begingroup$ @ayush. I describe the calculation in the third paragraph. In terms of actually calculating the resulting Chi Square p value, you can use an Excel formula. If I remember correctly it is either CHIDIST or CHITEST. $\endgroup$
    – Sympa
    Nov 7, 2010 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.