7
$\begingroup$

enter image description hereI am currently testing a binary logistic regression model (N=2000), examining the relationship between several independent variables (such as substance use -categorial-, gender-categorial-, self-control-likert-) and a binary/categorical dependent variable related to delinquency (no crime-at least one crime). However, I am facing a challenge with the Nagelkerke R-squared, which appears to be lower than expected... really really low. Despite all the independent variables showing statistical significance, the overall explanatory power of the model seems inadequate.

I used five independent variables in my analysis and would appreciate insights into why the Nagelkerke R-squared is low despite the significance of the individual predictors. Additionally, I am curious if there are alternative ways to improve the R-squared or if there are other indicators that could provide a more comprehensive assessment of the model's performance.

Any guidance or suggestions would be greatly appreciated. Thank you!

$\endgroup$
3
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Jan 8 at 19:13
  • $\begingroup$ Please tell us what "really, really low" means. Before you make any effort to reply, please see our thread at stats.stackexchange.com/questions/3559. $\endgroup$
    – whuber
    Commented Jan 8 at 20:49
  • 1
    $\begingroup$ @Julien That's not what the image states. The "0.001" is a termination criterion for successive changes in a numerical solution. The Nagelkerke $R^2$ statistics appear in the right hand column and range from $0.037$ to $0.088.$ $\endgroup$
    – whuber
    Commented Jan 9 at 14:56

3 Answers 3

12
$\begingroup$

You have a fairly large sample size of $2000$, meaning that you have the power to detect even small effects. What you're seeing can be interpreted as strong evidence of a weak relationship. I have likened this to the Princess and the Pea fairy tale where the princess has such sensitive skin that she can detect even the tiny pea under the mattress [1, 2 3].

It is typical to see this kind of behavior. For instance, in the paper I mention in this answer of mine, the performance was very modest despite statistical significance of the coefficients.

For instance, see the simulation below where a low Nagelkerke $R^2$ less than $0.03365$ is achieved despite each regression coefficient being significant at the $0.05$-level.

library(MLmetrics)
set.seed(2024)
N <- 2000
p <- 5
X <- matrix(runif(N*p, 0, 1), N, p)
B <- rep(0.7, p)
z <- X %*% B - 0.7
p <- 1/(1 + exp(-z))
y <- rbinom(N, 1, p)
model <- glm(y ~ X, family = binomial)
null  <- glm(y ~ 1, family = binomial)

# Nagelkerke R^2
#
nag_r2 <- function(L_model, L_null, n){
  
  numerator <- 1 - (L_null / L_model)^(2/n)
  denominator <- 1 - (L_null)^(2/n)
  return(numerator/denominator)
}

# Exponentiate the log-likelihood to get the likelihood
#
L_model <- exp(MLmetrics::LogLoss(predict(model, type = "response"), y))
L_null  <- exp(MLmetrics::LogLoss(predict(null,  type = "response"), y))

summary(model)
nag_r2(L_model, L_null, N)
$\endgroup$
7
$\begingroup$

p values are partly an effect of sample size. With a large sample size (and 2000 is pretty big) small effects can be significant.

For measures of fit for logistic regression see Allison, Measures of fit for Logistic Regression, presented and published at SUGI (SAS Users group) and available on the web. While Allison uses SAS a bit here, that's really not important. You should read the whole paper, but, briefly, he recommends

  1. Standardized Pearson Test
  2. Unweighted sum of squares
  3. Information matrix test
  4. Stukel's test.

He shows how to implement these in SAS, this might be used by someone to implement them in R or Python. Someone may have already done this.

$\endgroup$
4
$\begingroup$

The frequency distribution of Y was not provided. In the most information-rich case of 1000 in each of two groups, the effective sample size is $2000 \times \frac{1}{2} \times \frac{1}{2} \times 3 = 1500$. If the distribution is imbalanced, the effective sample size can be much smaller. For example if a fraction of 0.1 of the observations have Y=1, the effective sample size is only 540. The Cox-Snell and Nagelkerke pseudo $R^2$ using the apparent sample size and not the effective sample size in their calculations. As done with the R rms package lrm function, pseudo and adjusted pseudo $R^2$ calculations are done using the effective sample size. This is described here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.