I am fairly new in the field of Information retrieval. I have basic knowledge about machine learning. I understand the purpose of CV in the context of Machine learning. However, I've become a bit confused when I saw CV used in the context of Information retrieval.

Here, in this paper the authors said: "values of the free parameters are set using leave-one-out cross validation performed over queries, where MAP serves as the optimization criterion."

How to perform CV over queries?

Here is what I am thinking, we should split the queries (in the test collection) into 10-folds,

For i in 10:

  1. Using the training 90% part, we optimize the free parameter p (whatever the parameter is) for MAP (chose p that yield to the best MAP over queries)
  2. Test the chosen K against the testing part.

The Question is: After 10 iterations, we end up with 10 different values of P, what value should I use?


A very typical method to set parameters of a model is through maximum likelihood estimation; i.e., set the parameters to values that maximizes the likelihood of the observed data.

I presume that when the authors say they set the parameters of the model through cross-validation, they chose the values of the parameters (or, more likely given that the discuss using MAP estimation, the hyper-parameters) that the minimizes the estimated out-of-sample error during cross-validation, for some given loss function.

So I would guess that the authors are using cross-validation to the pick the hyper-parameters and then fitting the full data using those hyper parameters selected by cross-validation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.