2
$\begingroup$

Problem Setup: Let's say a bag contains 100 balls, 60 of which are red and 40 are black. 100 Balls are drawn at random from the bag (with replacement) and before each draw, you have to guess whether the drawn ball will be red or black.

Solution 1: You always guess that the drawn ball will be Red. This will result in an expected accuracy of 0.6

Solution 2: You flip a biased coin with $P(Heads) = 0.6$ and if the coin lands head, you guess that the drawn ball will be Red otherwise Black. Please note that the probability of 0.6 was chosen because the $P(red ball)$ in the bag is 0.6

Question for the Cross Validated Community: What is the expected accuracy for solution 2?

PS: This question was marked to be Closed because it was not clear what was being asked. Therefore, I have completely reworded the problem and the scenario and have explained it in as simpler way as possible.

Improve this question
$\endgroup$
9
  • $\begingroup$ You'll have to say a bit more about what you are trying to accomplish with this "assignment." If you already know which cases have cancer, it's not clear that there's any need for "assignment." $\endgroup$
    – EdM
    Commented Dec 25, 2017 at 19:58
  • $\begingroup$ @EdM: Generally there is a concept of training set and test set. Also one assumes that both training and test set follow the same distribution. P(cancer) = 0.6 was determined by looking at the training set and then for every case you see in the test set, you have to determine whether it is a case of cancer or not. So my "assignment" is for the cases in the test set. $\endgroup$
    – The Wanderer
    Commented Dec 25, 2017 at 20:27
  • $\begingroup$ @EdM: Let me also briefly introduce the concept of baselines. Baselines are meant to be some really simple way of accomplishing the task. One generally compares "fancy methods" with the performance of baseline methods. $\endgroup$
    – The Wanderer
    Commented Dec 25, 2017 at 20:30
  • $\begingroup$ Is there a reason you need to hard assign each case to cancer or not cancer? Often a better idea is to use a proper scoring rule loss function like log-loss, which measures how well the probability agrees with the class. It is really only in rare cases where accuracy is a reasonable evaluation metric. $\endgroup$
    – Matthew Drury
    Commented Dec 25, 2017 at 22:03
  • $\begingroup$ @MatthewDrury: Evaluation metric is not the main concern here. I am fine using Precision and Recall as well. My concern is, how do I measure that for the case of the probabilistic assignment? $\endgroup$
    – The Wanderer
    Commented Dec 25, 2017 at 22:46

1 Answer 1

Reset to default
2
$\begingroup$

You're combining two independent events, so their joint probability distributions can be split: $$ \begin{align} P(Correct) &= P(Red_{Ball},Red_{Coin}) + P(Black_{Ball},Black_{Coin}) \\ &= P(Red_{Ball})\cdot P(Red_{Coin}) + P(Black_{Ball})\cdot P(Black_{Coin}) \\ &= 0.6 \cdot 0.6 + 0.4\cdot 0.4 \\ &= 0.52 \end{align} $$

EDIT: To address what I think your original question may be about: Yes, consistently making the "correct" guess is better than randomly choosing the "correct" guess in proportion with the probability that that guess is correct. By only choosing red 60% of the time, you're making the "wrong" choice 40% of the time, as shown by the lower probability of 0.52 vs 0.6.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.