10 folds cross validation training sets

Question

So right now I have an okay understanding of 10 folds cross validation from reading and studying. I have a quick question about the training-set vs holdout set.

Let's say I'm doing 10 folds with 1 hold out on a linear regression model. I break my dataset into 10 pieces, I understand that 9 is used to train and 1 is used to test. Each of the 10 folds will be selected as the test set at some point.

My question is, how does 10 folds method determine the best parameter in the end? In case I wasn't clear, here's an example: so I understand that each fold is compared against the other 8 for the best fit parameters by running it with the validation set. Let's say the first time param A is selected, the second time around another fold is selected, let's call it param B. How does the method make a decision if param A or B is better?

Answer 1

Start with something simpler. Say you have a parameter that can many values and you want to see which value is best for that parameter. What would you do? You would loop through all the values for that parameter and try them out.

How do you try them out? This is where k-fold cross validation comes in. (It need not be 10 fold by the way). You split the training data into 10 folds and loop through the folds. For each fold, you train the model on the other 9 folds and then evaluate the model on the held out fold.

So, for each parameter value, you will have k (in your example, 10) values. You can average these 10 and this will give an evaluation of how that parameter value is.

This pseudo code might be clearer (in this case, I assume you are trying to minimize the error):

best_parameter_value <- Nothing 
best_parameter_score <- infinity 
for p in parameter_values:
    folds <- split data into k folds
    results_per_fold <- empty list
    for f in folds:
        model <- train model on the other k-1 folds with parameter p
        result <- evaluate model on f and get the error
        add result to results_per_fold
    average <- mean of results_per_fold
    if average < best_parameter_score:
        best_parameter_value = p 
Return best_parameter_value

Answer 2

how does 10 folds method determine the best parameter in the end?

Strictly speaking: it doesn't.

The best parameter is usually determined by comparing the performance (which may be measured by cross validation - or by other methods) of a number of models that were built with varying hyperparameters. The hyperparameter value that lead to the best performance is then selected.

Remember that by such a selection, you "use up" the performance estimate: it enters the model training that way and that is the reason why you need another performance measurement of the final model that is independent also of this selection procedure.

---

here's an example: so I understand that each fold is compared against the other 8 for the best fit parameters by running it with the validation set.

No. Typically all 10 folds are evaluated with the same parameter (s).

Let's say the first time param A is selected, the second time around another fold is selected, let's call it param B.

No. The fold is not a parameter to select. Hyperparameter and fold (or more precisely: your whole performance measurement scheme) are independent of each other.

How does the method make a decision if param A or B is better?

You can tabulate your performance measurements (loss function values, errors) as table of case against parameter value, e.g.

parameter value:   A    B
loss:
case 1             L1A  L1B    
case 2             L2A  L2B                
...            
case n             L1A  L1B                

If you evaluate both hyperparameter values with the same cross validation splits (and of course for the same cases), you can set up the comparison as a paired test. For example, for each case you calculate the difference LiA - LiB and test whether that is different from zero. (This is just an example, there are other tests such as rank tests and not paired tests as well)