I am developing a search engine system based on the vector space model, and I am confused on what approach I should take to evaluate the system.
My case is this:
- I have a set of indexed documents in pairs (di,ti) stored in a database. And I can calculate the similarities between a query and the set of indexed documents in the system using tf-idf weighting scheme and the cosine similarity measure and rank the document in descending order of the similarity score.
How do I evaluate the system using k-fold cross validation and the F1 measure? In this case what would my Training Set, Validation Set and Test Set be ? I am confused which dataset to split into these three. What data do I use for these three sets in terms of query terms, terms, documents and similarities ?
As I understand it, I will have to split the dataset into:
Test Set: A set of query terms
Training Set: A set of documents/term pairs
Validation Set: A set of similarity/document pairs ?
How can I combine the F1 Score with the K-Fold cross validation ?
Sorry if my question is ambiguous, but I don't completely understand these evaluation methods and I am new to all of these.
I implemented the system with plain python and django for interacting with the database, I am not using any libraries such as numpy, scikit learn.
I will try to explain my system further since I have not included all the details and that's why I am confused between the two.
I have another case where I have a set of tags for each of the document, so there is a set of tags assigned to those documents, by some users.
So my searching algorithm works for two different cases:
The first is as described above where the tf-idf weighting scheme is used and the cosine similarity measure to calculate the similarity between some queries and the documents.
The 2nd approach is: I calculate some weights based on the tagging behaviour on the documents, and I am adding these weights to the tf-idf weights i.e if the the term and tag is the same then I add their weights together, if the document doesn't have a term which is the same with the tag then the addition would be 0 + the tag/document weight (since there is no such term in the document term/document relation, and thus its term/document weight would be zero)
Now I want to evaluate the 1st and 2nd case and compare the rankings, so for the 2nd case I am assuming I can use k-fold ?
What approach would you suggest to evaluate the ranking of the results for such a scenario?