1
$\begingroup$

This is an abstract question, an attempt at roughly heuristicking an answer that would otherwise take significant cost and data to answer. If the "fuzziness" of the question makes it a bad fit for this site, please suggest how I can improve the question to be a better fit.

I'm trying to gather data for training a model by way of user rating trials, and I'm trying to figure out which of two rating systems, systems of assigning standardized ratings to entities based on repeated tests, will converge faster to an accurate score.

The first option is a Tinder-style rating system, where each trial consists of the user being presented with a single item, and rating it either "yes" or "no" in isolation. The second option is a pairwise comparison system, where each trial consists of the user being presented with two items, and choosing which of the two items is preferred.

Based on an arbitrarily large number of users performing an arbitrarily large number of trials, which of these two systems would be expected to converge more rapidly?

If you feel like being particularly ambitious, both of these options also offer a time component, so instead of each comparison providing one bit of information (yes/no for the first, and this/that for the second), each comparison could provide significantly more information in the form of how long it takes the user to decide. This fact may impact the choice of which rating system converges more quickly.

$\endgroup$
0
1
$\begingroup$

Your question is indeed too fuzzy in the sense that the answer essentially is "it depends". When you speak of "accuracy" and "efficiency" these concepts are all depended on the underlying model that you are assuming for the data. You could probably design a model that is somewhat appropriate such that either scenario is "better" by some metric.

e.g. Do we assume some latent univariate "quality" scale for each item? Do the raters have an equivalent latent "grumpy" scale (e.g. some raters just tend to not like any items, some like all items), or do we assume all these raters are equivalent in this regard? If not how do we define a zero-point in the latent item quality scale? Or perhaps we don't want a univariate scale but rather multivariate (some items are better in dimension 1, but others are better in dimension 2).

Defining the underlying model for how comparisons are done should be done first. This will often inform what the appropriate way to collect data is. Of course your model will be wrong in the end. The question is not how right or wrong it is, but how useful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.