K-fold cross validation procedure using 3 folds

Question

I'm writing a code for k-fold cross validation for lasso. But I am stuck at understanding it clearly.

The number of folds that I'm using is 3 (k=3)

I have a matrix X[20x15], I split it into two sets the training and the testing, X_train[13x15] and X_test[7x15] then I generated some data Y_train[13x1] and Y_test[7x1]

Then I generated a coefficient vector by using the training data.

I know that I have to introduce the Y_test and X_test and to compute the mse(mean square error) and this is the part which is not very clear what to do next. I know the mse is calculated using this formula:

I think the Yi will be the Y_train and the will be the X_train, but then do I have to do the same separately with the test data? Or do I have to use somehow the test data in the training?

Answer 1

You are supposed to split the data to 3 sets:$X_1,X_2,X_3$(and do the same with $Y$). Note that those are matrices. I will use small letters for individual samples. Then you need to train 3 models. The first model will use $X_1$ and $X_2$ as training data(by that I mean put those two sets in a single matrix). The second will use $X_1$ and $X_3$. The third will use $X_2$ and $X_3$. For all these, each time you also need to use the correct $Y$s.

Note that after this procedure you have 3 coefficient vectors. Let us denote your model that uses $\beta_i$ to make a prediction for $x_j$ with $y_j=f_i(x_j)$.

To calculate the MSE let me first define the error for a single sample: $L(y_j) = (y_j - f_i(x_j))$. Note: For each sample $x_j$ when you are calculating the error, you need to use the model that did not include $x_j$ in its training data. So for example $L(y_1)$ should use the model that included $X_2$ and $X_3$ as its training data. The other two models have already seen $x_1$ and $y_1$ during training so you are not going to get a 'fair' measure.

Then you calculate $MSE = \frac{1}{n}\sum_j^n[L(y_j)^2]$