Skip to main content
added 31 characters in body
Source Link
Christian Hennig
  • 28.3k
  • 35
  • 101

With 4 bottles there are only 24 possible rankings, so if you wanted to test at 5% level, the only possibility to reject random ranking would be to get the ranking exactly right (prob. 1/24). Testing at 1% level is impossible, there are simply not enough possibilities for outcomes. Note in particular that no test will give you a "probability that your answer is fully random"; the only thing a test can do is to see whether there is clear evidence against it. For finding the said probability you'd need a Bayesian approach and specify a prior probability for your ranking being "fully random" (and also prior probabilities for all kinds of other possibilities, which may be very tedious).

With more than four bottles you could use the Kendall tau distance between the rankings as loss. Chances are that at least for a low number of bottles one can compute precisely (by complete enumeration of all possibilities if all else fails) a p-value for a test that rejects randomness if your ranking is "too close" to the true one (which basically amounts to your idea but without simulation; if numbers become prohibitive, of course simulation is an option as in Stephan Kolassa's answer).

With 4 bottles there are only 24 possible rankings, so if you wanted to test at 5% level, the only possibility to reject random ranking would be to get the ranking exactly right (prob. 1/24). Testing at 1% level is impossible, there are simply not enough possibilities for outcomes. Note in particular that no test will give you a "probability that your answer is fully random"; the only thing a test can do is to see whether there is clear evidence against it. For finding the said probability you'd need a Bayesian approach and specify a prior probability for your ranking being "fully random" (and also prior probabilities for all kinds of other possibilities, which may be very tedious).

With more than four bottles you could use the Kendall tau distance between the rankings as loss. Chances are that at least for a low number of bottles one can compute precisely (by complete enumeration of all possibilities if all else fails) a p-value for a test that rejects randomness if your ranking is "too close" to the true one (which basically amounts to your idea but without simulation; if numbers become prohibitive, of course simulation is an option).

With 4 bottles there are only 24 possible rankings, so if you wanted to test at 5% level, the only possibility to reject random ranking would be to get the ranking exactly right (prob. 1/24). Testing at 1% level is impossible, there are simply not enough possibilities for outcomes. Note in particular that no test will give you a "probability that your answer is fully random"; the only thing a test can do is to see whether there is clear evidence against it. For finding the said probability you'd need a Bayesian approach and specify a prior probability for your ranking being "fully random" (and also prior probabilities for all kinds of other possibilities, which may be very tedious).

With more than four bottles you could use the Kendall tau distance between the rankings as loss. Chances are that at least for a low number of bottles one can compute precisely (by complete enumeration of all possibilities if all else fails) a p-value for a test that rejects randomness if your ranking is "too close" to the true one (which basically amounts to your idea but without simulation; if numbers become prohibitive, of course simulation is an option as in Stephan Kolassa's answer).

Source Link
Christian Hennig
  • 28.3k
  • 35
  • 101

With 4 bottles there are only 24 possible rankings, so if you wanted to test at 5% level, the only possibility to reject random ranking would be to get the ranking exactly right (prob. 1/24). Testing at 1% level is impossible, there are simply not enough possibilities for outcomes. Note in particular that no test will give you a "probability that your answer is fully random"; the only thing a test can do is to see whether there is clear evidence against it. For finding the said probability you'd need a Bayesian approach and specify a prior probability for your ranking being "fully random" (and also prior probabilities for all kinds of other possibilities, which may be very tedious).

With more than four bottles you could use the Kendall tau distance between the rankings as loss. Chances are that at least for a low number of bottles one can compute precisely (by complete enumeration of all possibilities if all else fails) a p-value for a test that rejects randomness if your ranking is "too close" to the true one (which basically amounts to your idea but without simulation; if numbers become prohibitive, of course simulation is an option).