Comparing a predictive model against a hypothetical model of random predictions Suppose there are three apples, two oranges, and one pear in a bag.
I put my hand in a bag, and based on my tactile sense, I guessed what fruit I will grab out of the bag.
My fruit predictions are the following:




Fruit
Prediction




Apple
Apple


Apple
Apple


Apple
Pear


Orange
Orange


Orange
Orange


Pear
Orange




I want to show that this prediction is better than random guesses. Thus, I make the following hypothetical model:

Knowing that there are three apples, two oranges, and one pear in the bag, randomly guess each fruit you grab.

By shuffling the list of fruits, I can make the "result" of this hypothetical model:




Fruit
Random prediction




Apple
Pear


Apple
Apple


Apple
Orange


Orange
Orange


Orange
Apple


Pear
Apple




Based on confusion matrix, I obtain the F1 score of the prediction of each fruit by my guesses and the random guesses.




Fruit
My guesses
Random guesses




Apple
0.8
0.333


Orange
0.8
0.5


Pear
NaN
NaN




Based on this comparison, I can say that my guesses are better than random guesses at predicting apples and oranges (but not necessarily pears).
Is this comparison statistically valid?
 A: I do not know whether the F1-measure can be used to compare with random guessing, but there are two other performance metrics specifically designed to compare to random guessing: Cohen's Kappa and Matthew's Correlation Coefficient.
You do not explain what you mean by an "better" prediction, but let us assume that you mean that the recognition rate $p$ is better than the recognition rate of random guessing $p_c$, i.e. $p-p_c>0$.
The recognition rate is estimated as the percentage of correct guesses, and $p_c$ can be estimated as
$$\hat{p}_c = \sum_i \hat{p}_i\hat{q}_i$$
where $\hat{p}_i$ is the proportion of samples with guessed index $i$ and $\hat{q}_i$ is the proportion of samples with true index $i$.
The question is how to normalize $\hat{p}-\hat{p}_c$ so that its value is restricted to the meaningful range $[-1,1]$. This is where Cohen's Kappa and MCC differ. It is
\begin{eqnarray*}\kappa & = & \frac{\hat{p}-\hat{p}_c}{1-\hat{p}_c} \\
MCC & = & \frac{\hat{p}-\hat{p}_c}{\sqrt{\left(1-\sum_i q_i^2\right)\left(1-\sum_i p_i^2\right)}}
\end{eqnarray*}
To test whether your guessing is statistically significantly better than random guessing at the significance level $\alpha$, you must construct $(1-\alpha)$ confidence intervals for $\kappa$ or MCC and check whether it contains zero (your Null hypothesis). Approximate confidence intervals for $\kappa$ have been given by Cohen (1960) and by Fleiss, Cohen, and Everitt (1969) (see, e.g., Reichenheim (2004) for the formulas). For MCC, I do not know of a closed formula for an approximate confidence interval, but you can use the bootstrap method to compute one, which is also a method to compute a confidence interval for $\kappa$ without relying on the asymptotic formulas.
