I don't know if I understood K-fold CV correctly. Here I will explain briefly what I am trying to do. I have a matrix X which I split into 3 folds, 2 training $X_1$, $X_2$ and one testing $X_3$:

enter image description here

Step 1

  • I concatenate $X_1$ and $X_2$ so I have $X_{1,2}$, then I generate a $Y_{1,2}$. I compute the $\hat{\beta_{1,2}}$ from the formula: $\hat{\beta_{1,2}}=(X^tX+\lambda I_p)^{-1}X^t Y$ and I compute the predicted $\hat{Y_{1,2}}=X_{1,2}\hat{\beta_{1,2}}$

step 2 I reshuffle. Then I have:

enter image description here

I do the same as in step 1, I compute $X_{1,3}$, $Y_{1,3}$, $\hat{\beta_{1,3}}$ and $\hat{Y_{1,3}}$

Then I reshuffle again and get:

enter image description here

And I compute again: $X_{2,3}$, $Y_{2,3}$, $\hat{\beta_{2,3}}$ and $\hat{Y_{2,3}}$

step 3 Computing the error (This is the part where I'm not quite sure if it is correct) The formula for the square loss is: $L(Y)=(Y-\hat{Y})^2$

Would then these be correct:




step 4 Calculating the mean of the above errors $CV = \frac{1}{n}\sum_j^n[L(Y)]$


Let's do a mini example with 6 samples:

Let $X_1$, $X_2$ and $X_3$ be three sets: $$ X_1 = \{x_1,x_2\}\\ X_2 = \{x_3,x_4\}\\ X_3 = \{x_5,x_6\}\\ $$

Each $x_i$ here is a vector. Judging from your previous question, this will be a 1x15 vector.

You also have: $$ Y_1 = \{y_1, y_2\}\\ Y_2 = \{y_3, y_4\}\\ Y_3 = \{y_5, y_6\}\\ $$

Those $y_i$ are single values.

Then you create three sets by merging those three folds in different combinations: $$ X_{1,2} = X_1 \cup X_2\\ X_{1,3} = X_1 \cup X_3\\ X_{2,3} = X_2 \cup X_3 $$

Once you train the three models you have $\beta_{1,2}$, $\beta_{1,3}$ and $\beta_{2,3}$.

Here is the important part. To calculate the predictions you do: $$ \hat{Y_1} = X_1\beta_{2,3} $$

Similarly for the other two: $$ \hat{Y_2} = X_2\beta_{1,3}\\ \hat{Y_3} = X_3\beta_{1,2} $$

Then your error is: $$ L(y_1) = (y_1 - \hat{y_1})^2\\ L(y_2) = (y_2 - \hat{y_2})^2\\ L(y_3) = (y_3 - \hat{y_3})^2\\ L(y_4) = (y_4 - \hat{y_4})^2\\ L(y_5) = (y_5 - \hat{y_5})^2\\ L(y_6) = (y_6 - \hat{y_6})^2\\ CV=\frac{1}{n}\sum_i[L(y_i)] $$

Note that $\hat{y_1}$ and $\hat{y_2}$ are samples from your matrix $\hat{Y_1}$. $\hat{y_3}$ and $\hat{y_4}$ are part of $\hat{Y_2}$. $\hat{y_5}$ and $\hat{y_6}$ is in $\hat{Y_3}$.


I find your notation a bit hard but I think I get it, additionally I think you are slightly off.

Once you have calculated $\hat \beta_{12}$ you want to use this to create predicted values for the values of $Y$ that were not in your training data, Lets call this $\hat Y_{3 \mid 1,2}$ . For clarity that is to say the predicted values of $Y_3$ after fitting a model using data from $X_{12}$ . You then calculate your error as $ (Y_3 - \hat Y_{3 \mid 1,2}) ^ 2$. Your last line is then correct that you then repeat this for all your folds of data and take the mean across all your predicted errors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.