How is Cross validation used with non-machine learning problems?

Question

I am fairly new in the field of Information retrieval. I have basic knowledge about machine learning. I understand the purpose of CV in the context of Machine learning. However, I've become a bit confused when I saw CV used in the context of Information retrieval.

Here, in this paper the authors said: "values of the free parameters are set using leave-one-out cross validation performed over queries, where MAP serves as the optimization criterion."

How to perform CV over queries?

Here is what I am thinking, we should split the queries (in the test collection) into 10-folds,

For i in 10:

1. Using the training 90% part, we optimize the free parameter p (whatever the parameter is) for MAP (chose p that yield to the best MAP over queries)
2. Test the chosen K against the testing part.

The Question is: After 10 iterations, we end up with 10 different values of P, what value should I use?

Answer 1

A very typical method to set parameters of a model is through maximum likelihood estimation; i.e., set the parameters to values that maximizes the likelihood of the observed data.

I presume that when the authors say they set the parameters of the model through cross-validation, they chose the values of the parameters (or, more likely given that the discuss using MAP estimation, the hyper-parameters) that the minimizes the estimated out-of-sample error during cross-validation, for some given loss function.

So I would guess that the authors are using cross-validation to the pick the hyper-parameters and then fitting the full data using those hyper parameters selected by cross-validation.

Answer 2

Fine tunning in information retrieval is a way to report the optimized performance of your model with free parameters.

I cite Learning Information Retrieval Functions and Parameters on Unlabeled Collections,

After k folds, the average of the performance of M on all the subsets is reported.

In this case,

$MAP_{opt} = \frac{1}{10}\sum_{i=1}^{10}MAP_i$,

where $MAP_i$ is the $MAP$ value in the $K_i$ test set with the optimal $P_i$.

But, if you need one "optimal" $P$ you can choose the $P_i$ most frecuent as Probabilistic and machine learning-based retrieval approaches for biomedical dataset retrieval.