I am fairly new in the field of Information retrieval. I have basic knowledge about machine learning. I understand the purpose of CV in the context of Machine learning. However, I've become a bit confused when I saw CV used in the context of Information retrieval.
Here, in this paper the authors said: "values of the free parameters are set using leave-one-out cross validation performed over queries, where MAP serves as the optimization criterion."
How to perform CV over queries?
Here is what I am thinking, we should split the queries (in the test collection) into 10-folds,
For i in 10:
- Using the training 90% part, we optimize the free parameter p (whatever the parameter is) for MAP (chose p that yield to the best MAP over queries)
- Test the chosen K against the testing part.
The Question is: After 10 iterations, we end up with 10 different values of P, what value should I use?