K-Fold Cross validation and F1 Measure Score for Document Retrieval using TF-IDF weighting and some customised weighting schemes

Question

I am developing a search engine system based on the vector space model, and I am confused on what approach I should take to evaluate the system.

My case is this:

1. I have a set of indexed documents in pairs (di,ti) stored in a database. And I can calculate the similarities between a query and the set of indexed documents in the system using tf-idf weighting scheme and the cosine similarity measure and rank the document in descending order of the similarity score.

How do I evaluate the system using k-fold cross validation and the F1 measure? In this case what would my Training Set, Validation Set and Test Set be ? I am confused which dataset to split into these three. What data do I use for these three sets in terms of query terms, terms, documents and similarities ?

As I understand it, I will have to split the dataset into:

Test Set: A set of query terms

Training Set: A set of documents/term pairs

Validation Set: A set of similarity/document pairs ?

How can I combine the F1 Score with the K-Fold cross validation ?

Sorry if my question is ambiguous, but I don't completely understand these evaluation methods and I am new to all of these.

I implemented the system with plain python and django for interacting with the database, I am not using any libraries such as numpy, scikit learn.

Update 1:

I will try to explain my system further since I have not included all the details and that's why I am confused between the two.

I have another case where I have a set of tags for each of the document, so there is a set of tags assigned to those documents, by some users.

So my searching algorithm works for two different cases:

1. The first is as described above where the tf-idf weighting scheme is used and the cosine similarity measure to calculate the similarity between some queries and the documents.

2. The 2nd approach is: I calculate some weights based on the tagging behaviour on the documents, and I am adding these weights to the tf-idf weights i.e if the the term and tag is the same then I add their weights together, if the document doesn't have a term which is the same with the tag then the addition would be 0 + the tag/document weight (since there is no such term in the document term/document relation, and thus its term/document weight would be zero)

   Now I want to evaluate the 1st and 2nd case and compare the rankings, so for the 2nd case I am assuming I can use k-fold ?

   What approach would you suggest to evaluate the ranking of the results for such a scenario?

Answer 1

First of all you are confusing two things:

• Classification task - where you are trying to model a mapping $\phi(X) \rightarrow \{ 1,..,K \}$. Example of such mapping would be building a model to find the main topic of the document from the set of $K$ predefined topics.
• Ranking - where you are building some query similarity measure $s$, and for given query $q$ return the list of objects sorted according to your function $s$.

In machine learning community, your problem is called learning to rank, and quality measures that you are refering (like f1-measure) are not applicable for such problems (directly, as those are metrics of classification). For ranking problems you should consider using one of the special measures, like for example:

• DCG and NDCG;
• Mean reciprocal rank;
• Kendall's tau
• Mean average precision (MAP);

In all cases you will need some kind of test set, where you provide system with some kind of "gold standard" - you need some set of your data with already selected "best" answers, so the system can be compared to it. Once you create such a set, with "expected answers" you can split it to the training/test set using K-folds and run one of the above metrics.

K-fold cross folding is also a method for hyperparameters selection using restricted set of labeled data, so it is completely independent from used metrics, you can use it to optimize some parameters of your model. If this is a case, you split your data into two parts: training/testing, then run K-fold on training so it is internally splited into training (for actual training) and validation (for "testing" the hyperparameter choice), and once you select the best parameters - you test it on the "test" set.

From your post I deduce, that your method actually does not require learning (it is designed by hand), so you can skip the "training set" and use only a "test set" (and "validation set" if you want to optimize some parameter).