I'm writing a code for k-fold cross validation for lasso. But I am stuck at understanding it clearly.

The number of folds that I'm using is 3 (k=3)

I have a matrix X[20x15], I split it into two sets the training and the testing, X_train[13x15] and X_test[7x15] then I generated some data Y_train[13x1] and Y_test[7x1]

Then I generated a coefficient vector by using the training data.

I know that I have to introduce the Y_test and X_test and to compute the mse(mean square error) and this is the part which is not very clear what to do next. I know the mse is calculated using this formula:

I think the Yi will be the Y_train and the will be the X_train, but then do I have to do the same separately with the test data? Or do I have to use somehow the test data in the training?


1 Answer 1


You are supposed to split the data to 3 sets:$X_1,X_2,X_3$(and do the same with $Y$). Note that those are matrices. I will use small letters for individual samples. Then you need to train 3 models. The first model will use $X_1$ and $X_2$ as training data(by that I mean put those two sets in a single matrix). The second will use $X_1$ and $X_3$. The third will use $X_2$ and $X_3$. For all these, each time you also need to use the correct $Y$s.

Note that after this procedure you have 3 coefficient vectors. Let us denote your model that uses $\beta_i$ to make a prediction for $x_j$ with $y_j=f_i(x_j)$.

To calculate the MSE let me first define the error for a single sample: $L(y_j) = (y_j - f_i(x_j))$. Note: For each sample $x_j$ when you are calculating the error, you need to use the model that did not include $x_j$ in its training data. So for example $L(y_1)$ should use the model that included $X_2$ and $X_3$ as its training data. The other two models have already seen $x_1$ and $y_1$ during training so you are not going to get a 'fair' measure.

Then you calculate $MSE = \frac{1}{n}\sum_j^n[L(y_j)^2]$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.