Why do my cross validation delta values and MSE calculation conclude very different model fits?

Question

My data:

My model:

mod <- glm(Y2/Y1 ~ Var_1, data = df, family = binomial, weights = Y1)

summary(mod) shows that my response variable declines significantly as Var_1 increases. However, in order to assess model fit, I conduct (1) cross-validation, and (2) calculate the Mean Standard Error (MSE).

# CV
boot::cv.glm(df, mod, K=8)
# MSE
actual = df$Var_1
pred = predict(mod)
MSE = mean((actual - pred)^2)
MSE

The delta values obtained from cv.glm = 0.004935280 & 0.004797322 The calculated MSE is 3012.686.

I was under the impression that a good model fit is indicated by low delta values and a low MSE value. So what am I doing wrong for these two methods to provide such different results?

Are there any other procedures I could do to assess how reliable my model is, given the very small dataset?

Answer 1

I think what you are referring to is similar to a deviance goodness of fit test. The deviance goodness of fit test tests the null hypothesis that the model you present is just as good as a model which perfectly predicts the data (I'm being very loose here, you should read up on the test yourself). In short, you want to fail to reject the null of this test. Because your data is grouped, you can perform the test as such

library(tidyverse)

y1 = c(194, 221, 230,239,233,242,257,250)
y2 = c(94,174,192,124,196,181,196,133)
v = c(83, 23,35,72,40,34,21,86)
d = tibble(y1, y2, v)

model = glm(cbind(y2, y1-y2) ~ v, family=binomial(), data=d)

dev = model$deviance
dgf = model$df.residual
pval = pchisq(dev, dgf, lower.tail = F)
pval
>>> 2.021497e-05

In this case, we would reject the null, meaning that our model may not fit the data very well (again, you need to read up on the test yourself).

I will come back later this evening and clean this answer up, I am just a little per-occupied right now.