0
$\begingroup$

Let's say we have the following samples:

enter image description here

So there are three labels $y=\left\{1,2,3\right\}$, three binary logistic regression 1-vs-rest classifiers have been learned with model parameters $\beta_1 = \begin{pmatrix} 1 & 1 & 2 \end{pmatrix}$, $\beta_2 = \begin{pmatrix} 0 & -3 & 2 \end{pmatrix}$, $\beta_3 = \begin{pmatrix} 1 & 1 & -4 \end{pmatrix}$.

How do you calculate the final misclassification rate / final error?


I think I can work with the normal loss function for logistic regression? I mean this one: $$L(\beta) = -\frac{1}{m}\sum_{i=1}^{m}cost(h_\beta(x^{(i)}), y)$$ where $h_\beta(x) = \sigma(\beta \cdot x)$ and $m$ means total amount of $(x_1,x_2)$ points thus $m=6$.

How ist $cost$ defined for this specific example with 3 classes? Like this is correct?

$cost_\beta(x,y)=\left\{\begin{matrix} h_{\beta_1}(x), \text{if } y=1\\ h_{\beta_2}(x), \text{if } y=2\\ h_{\beta_3}(x), \text{if } y=3 \end{matrix}\right.$

Are the things correct so far? Still I don't know how to use all these things to get a number / the error in the end. I hope someone can help me because this will probably be asked in some test soon and I like to be able to solve a few things :c

$\endgroup$
2
$\begingroup$

Accuracy is, always, the number of correct guesses out of the total number of guesses. If you guess the right category when there are two categories, you had an accurate prediction. If you guess the wrong category when there are ten categories, you had an inaccurate prediction. Think of it like your score on a multiple choice test.

However, Cross Validated supports the use of proper scoring rules rather than accuracy. Proper scoring rules aim to find the correct conditional probability. For many reason, this is more useful than just accuracy. One reason for this is that accuracy is sensitive to a threshold, while proper scoring rules are not.

Brier score and log loss (crossentropy) are two common proper scoring rules. Their equations are quite easy in the binary case, and they have extensions to multiclass problems.

Brier: $L(y,\hat{y})=\sum_{i=1}^N \big(y_i -\hat{y}_i\big)^2$

Log Loss: $L(y,\hat{y})=-\sum_{i=1}^N \bigg[ ylog(\hat{y}) + (1-\hat{y}) log(\hat{y})\bigg]$

Here, $\hat{y}$ represents probabilities, not categories.

$\endgroup$
5
  • $\begingroup$ Thank you very much for your answer! We aren't allowed to use calculators in the test / exam, so I think I won't be able to use the log loss function and I will be supposed to work with the accuracy instead. For this specific example, we have $6$ points in total where each of them could be predicted to $3$ different classes, where for one point the probability is $\frac{1}{3}$ to assign it to its correct class. So the probability to assign 1 point to a wrong class is $\frac{2}{3}$. Then the final error is given by $$(\frac{2}{3})^6 \approx 0.08779 \approx 8.78\%$$? $\endgroup$
    – eyesima
    Oct 13 '20 at 17:40
  • 1
    $\begingroup$ The final error is whatever it is. If you get six correct classifications, your accuracy is 100%; if you get zero correct classifications, your accuracy is 0%. $\endgroup$
    – Dave
    Oct 13 '20 at 18:13
  • $\begingroup$ Ohh okay. I don't really understand how I could for example get the probability for class $1$ e.g. $\hat{y_1}$ which I will need to use one of the two formulas you have posted above. I think $\hat{y_1}$ is given by first entry in the list multiplicated with $\beta_1$ and their result put into the Sigmoid function? $\endgroup$
    – eyesima
    Oct 13 '20 at 18:21
  • $\begingroup$ I have tried to calculate the prediction $\hat{y}$ for the first point / entry in the table like this: $\begin{pmatrix} 1\\ x_1\\ x_2 \end{pmatrix} \cdot \beta_1 = \begin{pmatrix} 1\\ 1\\ 2 \end{pmatrix} \cdot \begin{pmatrix} 1\\ 1\\ 2 \end{pmatrix} = 1+1+4=6$ Now I put this value into Sigmoid function $\sigma(6)=\frac{1}{1+e^{-6}} \approx 1$ So I end up with the prediction $\hat{y}=1$ for the very first entry in the table which must be wrong. I'm having a huge misunderstanding here on how the probability is calculated and I would be so much grateful if someone could $\endgroup$
    – eyesima
    Oct 13 '20 at 21:19
  • $\begingroup$ I suggest asking a separate question about how multinomial logistic regression works. $\endgroup$
    – Dave
    Oct 13 '20 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.