Adjust thresholds in multi-class classification I have trained a random forest classifier on a (highly-imbalanced) 3-class problem (A 1% of the data, B 96%, C 3%) and obtained probabilities for each of the three classes. 
Currently I assign an observation to the class with the highest probability.
Is there a possibility to adjust the thresholds so I can make a statement like "With maximal 1% probability we will be predicting B though it is actually A"? How can this be done? I really need to control for this type of error while other errors are not that important.
 A: I don't think it's correct to talk about probabilities in context of random forests, as random forest classifiers do not attempt to produce accurate probabilities (see Olson et al. ). It's better to view the output as some score in the interval [0,1]. 
In random forest classification, each class $c_i, i \in {1, ..., k}$ gets assigned a score $s_i$ such that $\sum{s_i} = 1$. The model outputs the label of the class $c_i$ where $s_i = max({s_1, ..., s_k})$. So in order to adjust the thresholds, you can weight the scores $s_i$ by some weights $w_i$, such that you output the label of class $c_i$ with $s_i^* = max(s_1^*, ..., s_k^*) = max(s_1 \times w_1, ..., s_k \times w_k)$. (If you want the $s_i^*$ to add up to $1$, you need to normalize them.)

I'm not sure how or if answering questions such as "With maximal 1% probability we will be predicting B though it is actually A" is possible for > 2 classes, but for two classes, e.g. A and B, you can approach this by formulating a hypothesis test. 
Given a sample X, in case of binary classification there are exactly two hypotheses in the universe. (1) X belongs to class A or (2) X belongs to class B. For simplicity, I will assume that the classes are balanced. 
Sample X corresponds to a score $s_x \in [0, 1]$ and $s_x$ follows a different distribution depending whether the true class of X is A or B. Lets say that the mean score of samples in class A is bigger than the mean score of samples in class B, i.e. $\mu_A > \mu_B$. 
Given your random forest model and a test set, you can calculate the empirical distribution of $s_x$ under A and $s_x$ under B. 
Now, say you observed a sample X with score $s_x$. What is the probability that this class belongs to class A? To answer this, you can simply calculate the lower tail of the score-distribution under class A for $s_x$, i.e. the percentage of sample in class A with a score <= $s_x$. 
The resulting p-value corresponds to the probability that a sample X with score $s_x$ or lower truly belongs to the class A. You can do the same for class B by calculating the upper tail. 
Similarly, for a given significance level $\alpha$, e.g. $\alpha = 0.01$, you can calculate a score $s_\alpha$ such that the chance that a sample X with score $s_x <= s_\alpha$ belongs to class A is less than $1\%$. 
You see, since there are only two possible lables for a sample X, i.e. A or B, you can formulate a hypothesis test $H_0: X \in A, H_a: X \in B$ or vice verse. 
Now, in case you have >2 classes, this is no longer possible, since you can only reject a given hypothesis, i.e. that a sample X belongs to A. 
In in the binary case, rejecting A corresponds to accepting B, since there are only two possible outcomes, but for e.g. 3 classes, rejecting A corresponds to accepting B and C!
Keep in mind that the procedure above only works for balanced classes. You can possibly extend this approach for the imbalanced case, but it will likely be more complicated and I just wanted to outline a possible general approach. 
