In a project I'm participating in, the Nagelkerke R2 value from a logistic regression model is used to compare the performance of a set of scores in explaining some binary data. The regression is simply logic_vector = betas*scores. The scores are produced in a variety of ways, so the models can't really be considered nested. The same samples are always used, N=1227. Now if i have the Nagelkerke R2 values of two of the models and one is higher than the other, is there some way I can calculate if it is significantly higher? Example: if the R2 values I'm comparing are 0.1 and 0.8, it intuitively seems they are significatly different and the second set of scores is better since it explains more. Compare this to two sets of scores with R2 0.41 and 0.42 where it intuitively seems that the difference is random. Is there some method for doing this? I have looked at F change, Fischers transformation and bootstrapping, but since this logistic regression is just an evaluation of a score, i.e. there is only ever one explanatory variable, I can't determine if they are appropriate. P-values from the regression and AIC values are available.
Thanks for the answer. The problem is I can't do logic_vector = score1 + score2 + score3 ... since I dont have access to all scores. This also aswers Peters question above. I have the scores for all the models I have made, so with those I could do comparison with a full model, but the most important part is to compare with older models. What I basically have is a list of R2 scores from a bunch of models, p-values from the regression and AIC values. And I know the sample size is always 1227. In my mind it is not clear if AIC values are comparable between non-nested models, or if this whole idea of using the R2 value for comparison in this way is legitimate. So given a list of R2 values alone, is it possible to determine significant differences?
