GLM - overfitting [closed]

Question

I'm fitting a GLM in R with aorund 20k data points. Using around 8-10 parameters which I thought would be significant I am able to get a near perfect fit to the data. By a perfect fit, I mean comparing the actual response compared with the predicted value for each factor I've fitted.

In cases like this, what is the approach here to get an accurate model? Should I ignore the non significant factors? Is there simply not enough data to get an accurate prediction?

The model is a Poisson glm with a log link function.

Answer 1

The difficulty in answering your question is that the scenario you describe is essentially not possible. Except in special degenerate cases, it is not possible to use just 10 linear model parameters to get a perfect fit to a vector of 20,0000 values. This would be so for any GLM model, not just for a Poisson regression with log-link.

So one of two things must be true. Either you don't really have a perfect fit or the data is degenerate in some way, for example almost all zero.

Do you really have just 8-10 parameters, or do you perhaps mean 8-10 terms in your linear model? If the terms are factors, then each term could correspond to more than one parameter.

Do you really have a perfect fit? A perfect fit would mean that the residual deviance is zero. You can see the residual deviance if you obtain a summary

summary(fit)

or an analysis of deviance table

anova(fit)

for your fitted GLM model.

You say that you have compared the response to the "predicted value for each factor" but there is no such thing as a predicted value for a factor. You need to compare the responses to the fitted values from your GLM model, and those depend on all the factors at once.

Without seeing your R code or any of the actual R output, it's hard to say anything more.