I know about $r^2$ tells you about the amount of variation that can be explained by the predictor variables. I have run a model in which the rsquare has value 0.3010 but has false positive rate of around 15.60%. So this model which is logit in nature predicts 84% cases right. I want to know two things:
1) Is this false positive rate significant, i.e. is this prediction good enough?
2) Does it matter if I $r^2$ is as low as 0.3 if my model is good in prediction as we know that $r^2$ is a value used for explaning and not predicting?
Tell me more
×
Cross Validated is a question and answer site for
statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
|
|
|||||||||||||||||||
|
|
First, no model is perfect unless it is over-fit. So, your false positive rate is not unusual. Is a false positive rate of 16% good or bad? If it is lower than the natural proportion in the data it is OK. If it is not it is really bad. The key is by how much your model reduces the error rate. That's measured in several ways as mentioned in the second paragraph. Second, R Square is not a very well accepted measure of a model's robustness and explanatory power for Logit regression. I would not rely solely on it. There are other R Square measures earmarked for Logit regression. They include Pseudo R Square, Aldrich and Nelson R Square, and McFadden R Square. They all measure the reduction in error as captured by the -2LL calculation between the Baseline model that simply assumes that the probability for all observations is the average probability or the average proportion (if all values are 1 or 0). Another answer also mentions the Cox & Snell R Square and the Nagelkerke R Square. I suspect those are other variants on the same theme focusing on the model's error reduction. I would re-frame the investigation on the robustness of your model by focusing on the mentioned different R Square measures. I would also calculate the Chi Square p value for the whole model. The difference in the -2LL between the baseline model and the model is a Chi Square value. The number of variables gives you the DF. The resulting Chi Square p value gives you a measure of statistical significance for the whole model (probability that results were due to randomness). Maybe even more importantly, I would focus on the statistical significance of the independent variables you are using. The Wald statistics (really another Chi Square test) fits that purpose well. In this case, a variable regressed coefficient divided by the coefficient standard error gives you the Wald value. DF is always 1. And, the resulting p value conveys the statistical significance of this variable. With the above, you can then remove or add variables and attempt to improve your model. |
|||||||||
|
