0
$\begingroup$

I am fairly new in the field of Information retrieval. I have basic knowledge about machine learning. I understand the purpose of CV in the context of Machine learning. However, I've become a bit confused when I saw CV used in the context of Information retrieval.

Here, in this paper the authors said: "values of the free parameters are set using leave-one-out cross validation performed over queries, where MAP serves as the optimization criterion."

How to perform CV over queries?

Here is what I am thinking, we should split the queries (in the test collection) into 10-folds,

For i in 10:

  1. Using the training 90% part, we optimize the free parameter p (whatever the parameter is) for MAP (chose p that yield to the best MAP over queries)
  2. Test the chosen K against the testing part.

The Question is: After 10 iterations, we end up with 10 different values of P, what value should I use?

$\endgroup$

2 Answers 2

1
$\begingroup$

A very typical method to set parameters of a model is through maximum likelihood estimation; i.e., set the parameters to values that maximizes the likelihood of the observed data.

I presume that when the authors say they set the parameters of the model through cross-validation, they chose the values of the parameters (or, more likely given that the discuss using MAP estimation, the hyper-parameters) that the minimizes the estimated out-of-sample error during cross-validation, for some given loss function.

So I would guess that the authors are using cross-validation to the pick the hyper-parameters and then fitting the full data using those hyper parameters selected by cross-validation.

$\endgroup$
0
$\begingroup$

Fine tunning in information retrieval is a way to report the optimized performance of your model with free parameters.

I cite Learning Information Retrieval Functions and Parameters on Unlabeled Collections,

After k folds, the average of the performance of M on all the subsets is reported.

In this case,

$MAP_{opt} = \frac{1}{10}\sum_{i=1}^{10}MAP_i$,

where $MAP_i$ is the $MAP$ value in the $K_i$ test set with the optimal $P_i$.

But, if you need one "optimal" $P$ you can choose the $P_i$ most frecuent as Probabilistic and machine learning-based retrieval approaches for biomedical dataset retrieval.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.