2
$\begingroup$

Let's say I have three search engine e.g. search engine A, search engine B and search engine C.

Each search engine is given a set of queries Q (e.g. apple,banana,carrot....), this set Q remains the same for each engine.

Each engine then provides a set of results for each query in the form a SERP (search engine results page).This means for every query (q) we have three search engine results pages (SERPs).

We have recruited 4 users lets say, and each user is given multiple sets of queries and their paired SERPs, and asked to rank each SERP for a query all of the queries in terms of relevance on a scale of 1-4 (1 being not relevant, 4 being very relevant).

So user one will see and rank accordingly:

query 1(apple), SERP A (relevance 1), SERP B(relevance 1), SERP C(relevance 4)
query 2(banana), SERP A (relevance 2), SERP B(relevance 2), SERP C(relevance 4)
query 3(carrot), SERP A (relevance 3), SERP B(relevance 2), SERP C(relevance 3)

User two will see and rank as follows:

query 1(apple), SERP A(relevance 1), SERP B(relevance 2), SERP C(relevance 3)
query 2(banana), SERP A(relevance 1), SERP B(relevance 2), SERP C(relevance 4)
query 3(carrot), SERP A(relevance 3), SERP B(relevance 2), SERP C(relevance 3)

User three will see and rank as follows:

query 1(apple), SERP A(relevance 1), SERP B(relevance 2), SERP C(relevance 3)
query 2(banana), SERP A(relevance 2), SERP B(relevance 2), SERP C(relevance 3)
query 3(carrot), SERP A(relevance 3), SERP B(relevance 3), SERP C(relevance 4)

I was wondering if there was a measure that I could use to say which search engine produced the most relevant search results over a set of queries, based on the the scores the users gave. Would Chi-squared or an analysis of variance (ANOVA) be appropriate?

At a quick glance if this was A/B split testing I guess I would say results generated by search engine C provided better results than A or B, whilst B provided better than A. Am I correct in saying this and what would be the best to test to formally write this down??

$\endgroup$
0
$\begingroup$

It looks like a Friedman Test would be most suitable if I average all scores for each user for each search engine.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.