Suppose I have a data set with a categorical variable $Y$ (with possible values $j=1,\dots,J$) and another variable $X$. I wish to classify $Y$ based on the information in $X$.

For simplicity, suppose I also know the shape of the model that generated the data, though I do not know the parameter values; I will estimate them. E.g., this could be a (multinomial) logistic regression.

I am facing a loss function $l(\hat y,y)$ where $\hat y$ is the predicted class and $y$ is the actual/true class. $l$ can be represented by a matrix $\mathbf{L}$ where rows correspond to actual classes and columns to predicted ones. Each off-diagonal cell $l_{ij}$ ($i\neq j$) of $\mathbf{L}$ contains the loss associated with the specific misclassification.

Steps taken
I fit a (multinomial) logistic regression to the data. Given the fitted coefficients and a new data point with a known $X$ value $x_0$ but an unknown $Y$ value $y_0$, I obtain the fitted class probabilities $\hat p_0$ (a vector). I wish to classify $y_0$, i.e. obtain $\hat y_0$ so as to minimize the expected loss.

What is the optimal classification rule in this setting?
Could you also recommend a textbook chapter on the topic?

My problem is somewhat similar to Classification optimal decisions considering a loss function but my setting is frequentist and I do not have the prior distribution of classes (or class prevalence) available.

  • $\begingroup$ I know this is pretty basic, but I am struggling to find good textbook material on the topic. The few machine learning textbooks I looked at seem to stop short of optimal decision making (tailoring the decision to the loss function, given the rest). $\endgroup$ – Richard Hardy Oct 2 '19 at 7:46

I have considered the following decision rule:

  1. Calculate the vector of "estimated expected loss" (denote it by $\widehat{el}(\mathbf{L},\hat p_0)$) consisting of elements corresponding to classification decisions $j=1,\dots,J$ (assigning class $j$ to $\hat y$) as $\widehat{el}(\mathbf{L},\hat p_0):=\mathbf{L}^\top \hat p_0$.
  2. Find the minimal element of the vector and select the corresponding predicted class.

I wonder if this is optimal.

  • $\begingroup$ Since I do not know (multinomial) logistic regression well, I am not really sure what the fitted probabilities $\hat p_0$ are and whether they constitute a sensible estimate of true probabilities (hence the quotation marks for "estimated expected loss"). I have a gut feeling I might be ignoring the prevalence of each class (sort of the prior which I do not have) and perhaps implicitly assuming uniform class prevalence or something like that. $\endgroup$ – Richard Hardy Oct 2 '19 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.