I have the following data

'data.frame':   768 obs. of  5 variables:
 $ PIANTA     : chr  "C-1-R1-1" "C-1-R1-1" "C-1-R1-2" "C-1-R1-2" ...
 $ Trattamento: Factor w/ 4 levels "Controllo","Lidar",..: 1 1 1 1 1 1 1 1 1 1 ...
 $ Blocco     : Factor w/ 2 levels "1","2": 1 1 1 1 1 1 1 1 1 1 ...
 $ Replica    : chr  "R1" "R1" "R1" "R1" ...
 $ Risposta   : num  0 1 0 1 0 3 2 3 2 4 ...

I have a total of 768 observations. I would like to test whether the treatment (Trattamento) has a significant effect respect to my response variable (Risposta) and include the possible role of a blocking factor (Blocco). The response variable is numeric (ranging from 0 to 9) and assumes the value 0 for more than 400 observations.

Therefore I opted to use a negative binomial regression model using the following R code:

glm.nb(formula = Risposta ~ Trattamento + Blocco, data = data, 
    init.theta = 0.545484172, link = log)

                    Estimate Std. Error z value Pr(>|z|)    
(Intercept)          0.26016    0.13268   1.961 0.049904 *  
TrattamentoLidar    -0.60387    0.17507  -3.449 0.000562 ***
TrattamentoRecupero -0.86591    0.18135  -4.775  1.8e-06 ***
TrattamentoStandard -0.35299    0.17027  -2.073 0.038159 *  
Blocco2             -0.03355    0.12671  -0.265 0.791162    
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for Negative Binomial(0.5455) family taken to be 1)

    Null deviance: 677.61  on 767  degrees of freedom
Residual deviance: 651.46  on 763  degrees of freedom
AIC: 1916

Number of Fisher Scoring iterations: 1

              Theta:  0.5455 
          Std. Err.:  0.0654 

 2 x log-likelihood:  -1903.9550

I see that all treatments are significant with respect to the intercept (control) and that the block has no significant effect. I wanted to test the predictive capacities of the model and I have used the following code:

predicted <- predict(model1, type = "response")

predictions <- data.frame(Osservato = data$Risposta, Predetto = predicted)

plot(predictions$Osservato, predictions$Predetto, 
     xlab = "Observed", ylab = "Predicted",
     main = "")

abline(a = 0, b = 1, col = "red")

legend("topleft", legend = "Linea di riferimento", col = "red", lty = 1)

I obtain the following graph of predicted versus observed:

enter image description here

From the graph, it seems that the model is not good for predicting the values of my response variable. Given that, and considering that predicting values is not the scope of my study, can I still rely on the significance values obtained by the model?


1 Answer 1


Statistical significance and predictive power are not totally related. Take for example this statistically significant but nonetheless extremely poor fit. It yields pretty terrible predictions given it tries to fit a linear relationship to nonlinear data:

#### Sim Data ####
x <- runif(1000,0,8)
y <- cos(x) + rnorm(1000,sd=.1)

#### Fit and Summarize ####
fit <- lm(y ~ x)

#### Plot ####
     main = "Poor OLS Fit")

#### Add Regression Line ####

enter image description here

Since you mention there being a ton of zeroes, you could consider instead something like zero-inflated regression. Check that it is indeed a problem (you can use the check_zeroinflation function from this package). Since you are using R, there is a brief tutorial here, though I would recommend reading Hilbe's book on negative binomial regression for a more in-depth but practical source of learning.

  • $\begingroup$ Thank you @Shawn for the information. Basically, you are telling me that the significances found by the glm can still be valid even though the model has no good predictive capacities. I will try also the zero inflation $\endgroup$
    – GiorgioS
    Feb 9 at 8:56
  • $\begingroup$ They can be, but only if the model is correctly specified. The problem with zero-inflated data is that a regular regression will predict too few zeroes than there are actual observed zeroes in the data, and will thus have weak predictive power. That is of course if the zero-inflation is severe enough to warrant concern. $\endgroup$ Feb 9 at 11:55
  • $\begingroup$ What you mean with a model correctly specified? Which diagnostic shall I run to better understand if the model well represents significance levels among treatments? $\endgroup$
    – GiorgioS
    Feb 9 at 12:04
  • $\begingroup$ What I mean is when you have a case like the one I illustrated above with the simulated data, the $p$ values are meaningless. And statistical significance in any case shouldn't be a barometer of success for a model. The primary goal is that you have fit the data correctly, regardless of low or high $p$ values. $\endgroup$ Feb 9 at 12:06
  • $\begingroup$ Thank you. This is quite clear in the case you illustrated as you were dealing with continuous variables. But in my case I have a count (only integer) variable and an independent categorical variable. How can I check model fit in this case? $\endgroup$
    – GiorgioS
    Feb 9 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.