I have some questions about best practices in information retrieval (unsupervised) vs learning to rank evaluation.
How necessary in a train-validation-test or cross-validation scenario? is it usually considered? why/why not?
a) train and validation (indexed) documents are disjoint: no document is indexed in train AND validation datasets (like Yahoo LTR challenge, but not in MQ2008)
b) consider a set of queries without relevant results where the expected system result is "no results"
c) Is it correct to compare traditional IR vs LTOR in the following way? IR: index test documents, run queries and check relevance LTOR:
1) Optimize hyperparameters training LTOR model with train data and evaluating with validation data 2) Index test documents 3) Top-k pooling test documents using from index using test queries, and train LTOR model with test (with hyperparameters of step 1) with queries/documents results 4) Re-rank documents using the fitted model in 3)
And finally, evaluate metrics on IR and LTOR results. Is there a better way?
Any reference is welcome!
References: In traditional Information Retrieval evaluation, I read An Introduction to Information Retrieval (Manning, Prabhakar, Schütze), where evaluation takes place considering: Information retrieval: 1. A document collection 2. A test suite of information needs, expressible as queries 3. A set of relevance judgements, ..., relevant or nonrelevant for each query-document pair
On other hand, taking Learning to Rank references:
- Yahoo LTR challenge take queries and documents partitions
queries: train, validation, test documents: train, validation, test
- Microsoft LETOR 4.0 MQ2008
Fold Training.txt Validation.txt Test.txt
Fold1 S1, S2, S3 S4 S5
Fold2 S2, S3, S4 S5 S1
Fold3 S3, S4, S5 S1 S2
Fold4 S4, S5, S1 S2 S3
Fold5 S5, S1, S2 S3 S4