Distribution of GLM coefficient estimates when cross-validating I am cross-validating a Poisson regression model to investigate its accuracy through randomly defined training/validation sets across 1,000 iterations.
When looking at the distribution of the coefficient estimates of the variables used in the training set across the 1,000 iterations interesting patterns emerge: some coefficients show huge variablility across iterations while others less.
I was wondering what does this mean? 
 A: If the coefficients show huge variability, it means that the model depends too much on the data, and it is not good. It can be caused by two possibilities:


*

*Overfitting: you are including the variables that you shouldn't. These variables have nothing to do with the DP. So the coefficients estimated of these variables depend to much on the data.

*Underfitting: you are not including enough necessary variables. So, by changing the training data set, the coefficients of included variables need to be adjusted in order to capture the signal missed by the variables not included in the model in the way that it should not be. 


Demonstration by simulation:
Constructing the data base:
Here I build a database with one DP and 10 IPs. The DP depends on only 3 variables: $X_2, X_4, X_7$ and a gaussian noise as follows:
#specify the number of IP and number of observations
IP_nb=10
obs=10000

X = matrix(0,nrow = obs,ncol = IP_nb)
for (i in 2:ncol(X)){
  X[,i] = runif(nrow(X))
}

epsilon = rnorm(nrow(X),0,1)
X[,1] = 1-X[,2] + 3*X[,4] + 5*X[,7]+epsilon

dat=data.frame(X)

The data base consists of 1 DP and 10 variables to be selected.
The first scenario:
the variable $X_9$ is included in the model (it should not be)
> set.seed(1)
> n=100
> alpha = 0.5
> cof1 = rep(0,n)
> for (i in 1:n){
+ d = sort(sample(nrow(dat), nrow(dat)*alpha))
+ train<-dat[d,]
+ test<-dat[-d,]
+ mod=lm(X1~X2+X4+X7+X9,dat=train)
+ cof1[i] = mod$coefficients[5]
+ }
> cof1
> hist(cof1/mean(cof1))


The coefficient for the variable $X9$ (which should not be in the regression) varies way too much as you can see on the graph.
In the second scenario where we missed some varia (underfitting)
[![set.seed(1)
n=100
alpha = 0.5
cof1 = rep(0,n)
for (i in 1:n){
d = sort(sample(nrow(dat), nrow(dat)*alpha))
train<-dat\[d,\]
test<-dat\[-d,\]
mod=lm(X1~X2,dat=train)
cof1\[i\] = mod$coefficients\[2\]
}
cof1
hist(cof1/mean(cof1))][2]][2]

The variation is not as bad as in the first case, but it is still big.

