set.seed(1)
n<-11
x1<-1:n
x2<-rnorm(n)
x3<-1/20*rnorm(n,x1,x1/7)+x2/5
cor(x2,x3) ;cor(x1,x3)
y<-1+5*x1+6*x2+2*x3+rnorm(n)
x4<-runif(n)
x5<-runif(n) ; x6<-runif(n) ; x7<-runif(n)
x8<-runif(n) ; x9<-runif(n) ; x10<-runif(n)
train<-data.frame(y=y,x1=x1,x2=x2,x3=x3,x4=x4,x5=x5,
x6=x6,x7=x7,x8=x8,x9=x9,x10=x10)
n<-11
x1<-1:n
x2<-rnorm(n)
x3<-1/20*rnorm(n,x1,x1/7)+x2/5
cor(x2,x3) ;cor(x1,x3)
y<-1+5*x1+6*x2+2*x3+rnorm(n)
x4<-runif(n)
x5<-runif(n) ; x6<-runif(n) ; x7<-runif(n)
x8<-runif(n) ; x9<-runif(n) ; x10<-runif(n)
test<-data.frame(y=y,x1=x1,x2=x2,x3=x3,x4=x4,x5=x5,
x6=x6,x7=x7,x8=x8,x9=x9,x10=x10)
I just created a train set and a test set in order to understand how the comparison of them can help me to find which model to fit.
You can see from my construction that only $x_1$, $x_2$ and $x_3$ are important to $y$. As a matter of fact, I claim that at most two of them are important to $y$, since $x_3$ is highly correlated with $x_1$ and $x_2$.
So I claim for example that the model
m1<-lm(y~x1+x2,data=train)
is much better than the models
m2<-lm(y~x1+x2+x3,data=train)
m3<-lm(y~x1+x2+x3+x4+x5+x6+x7+x8+x9,data=train)
m4<-lm(y~.,data=train)
and that $m_3$ is over-parameterized.
How can I use the test set and the train set to confirm these two claims? I have read similar questions online and some answers which used the MSE (mean squared error) and the predictive power of a model, but I understood none of them and I don't know how to apply them to solve this problem. How can I use them in R to answer my question?
And lastly, what is actually the out-of-sample predictive power of a model? Can it be measured with some way? What is the predictive power of the models $m_1$ and $m_3$? Which of these two models has a greater predictive power?