0
$\begingroup$

Suppose you have a set of objects and you have to guess whether each is A or B. Each object has a known probability of being A and a known probability of being B.

  1. How would you evaluate how "well" you are guessing when you go down the list of objects and assign each A or B.

  2. How would you evaluate how well you are guessing when you go down the list of objects and assign each a probability of being A or B.

Thanks for your help!

$\endgroup$
3
  • 2
    $\begingroup$ Hi, these are some rather standard starting concepts in decision theory or probability. Is this a homework question or practice problem? If so, please tag as self-study. What do you think the answers to your questions are, and where are you stuck? $\endgroup$ May 12, 2021 at 2:30
  • $\begingroup$ Do you have any features of A and B that you are using to make your guess? $\endgroup$
    – jerlich
    May 12, 2021 at 2:31
  • 1
    $\begingroup$ It's neither for homework or a practice problem - I'm trying to evaluate how well my friends and I are at predicting certain things in some games we play. is there an appropriate tag for this use case? $\endgroup$
    – Cloner5942
    May 12, 2021 at 2:43

1 Answer 1

0
$\begingroup$

Fun question.

In the first question, you are labelling objects as either A or B, making a definitive choice. If you do this for a sequence of objects, you can construct what is known as a confusion matrix

           Actually A Actually B
Assigned A          a          b
Assigned B          c          d

From this matrix, we can compute a whole host of metrics. Accuracy is probably the one people think of most often, but sensitivity and specificity are two other measures of how well one can guess.

In the second question, you would be assigning probabilities (p% this object is class A, q% this object is class B, etc). In that case, calibration of your probabilities is a measure you would want to examine. In brief, say you had 20 objects which you p[predicted all have 95% probability of being class A. If you are well calibrated for predicting the probability of class membership, then 19 of those 20 should actually belong to class A (that is 95% of the 20 actually belong to class A).

As Arya mentions, these are quite standard topics in the realm of classification and probabilistic modelling respectively. I will leave you to understand them further.

$\endgroup$
9
  • $\begingroup$ Thanks for your help! Will do some reading on the concepts you referenced. :) $\endgroup$
    – Cloner5942
    May 12, 2021 at 2:47
  • $\begingroup$ I'm reading about confusion matrices and I have a follow up question: Suppose you objects x and y, both class A. In one scenario from the POV of the guesser x has 90% chance of being A and y has 80% chance of being A. In another (also from the POV of the guesser) x and y both have 50% chance of being A. Is there a method where the guesser still guesses in a binary way but we can distinguish between the two? For example using the example someone that guesses A, A in scenario two would have done a "better" job than the person who guessed A, A in scenario 1. Thanks for your help! $\endgroup$
    – Cloner5942
    May 12, 2021 at 3:35
  • $\begingroup$ @Cloner5942 I'm not sure I understand. You're going to have to make your question more concrete. $\endgroup$ May 12, 2021 at 3:56
  • $\begingroup$ Suppose an object is in class A. Before making their guesses, guesser one's has information saying that the object has a 30% chance of being in class A and guesser two has information saying that the object is 90% in class A. Intuitively if both guess that the object was in class A I would think that the first guesser would be classified as a "better guesser", especially if we repeat the scenario a bunch. Is there some tool or technique to describe or formalize this? Sorry if still not clear. $\endgroup$
    – Cloner5942
    May 12, 2021 at 4:06
  • $\begingroup$ Here's an example that might help clarify Sam has to guess whether each of 100 objects is in class A or B. He has a list of probabilities each is in A, something of the form (50, 20, ...). He guesses. The next day Sam has to guess again, this time with 80 objects. He has another list of probabilities each is in A, maybe (25, 40, ...). He guesses. How do we analyze which day he did better on? How do we analyze how he's done overall? My understanding is confusion matrices don't care about the set of probabilities the guesser is working with but just looks at whether he's right or not. $\endgroup$
    – Cloner5942
    May 12, 2021 at 4:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.