Probability Density Function and Maximum Likelihood Estimation for Multinomial Logistic Regression and GMM

Question

I have some confusion about a few very basic concepts and terminology.

Let's assume we have two models for classification, a multinomial logistic regression (MLR) model and a GMM classifier. I'm not sure if "GMM classifier" is a correct term, but I mean that we fit a Gaussian distribution to each class, and to classify a new sample, we choose the class that the new sample fits the most.

For the MLR model, we have

$$ p(y_i=k|x_i, w_k)=\frac{\exp \lbrace w_k^Tx_i\rbrace}{\sum_{j=1}^K \exp \lbrace w_j^T x_i \rbrace} ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~(1) $$

where $y_i$ is the label for data point $x_i$, $K$ is the total number of classes, and $w_k$ is the parameters of class $k$.

My questions:

1. Is "GMM classifier" a thing? Or is what I described called maximum likelihood classifier, or something else?
2. What is the difference between the two that makes GMM (the thing that I described above) a mixture model while MLR is not? Or, is MLR also a mixture model?
3. Is (1) a probability density function?
4. In the likelihood of the parameters given in (1), what is the distribution of variables X and Y? Are they continuous?
5. If this is a continuous distribution and a pdf, how does it give probabilities for a specific values? For continuous variables, the pdf is zero at a specific point, and evaluating a pdf gives the value of the pdf, not the probability, right?
6. What about the case of GMM? By just evaluating the pdf for each class, we get the value of the likelihood or the value of pdf (I suppose the likelihood is the pdf here), and not the probability. How can we choose the class giving the highest probability? Do we need to integrate the pdf?
7. In the GMM case, the likelihood would be written as $p(x_i|y_i, \mu_k, \sigma_k)$. It is still a function of parameters, but $x_i$ and $y_i$ changed places. And we can define priors over parameters and labels, and to classify new data points we can maximize the posterior: $p(y_i|x_i, \mu_k, \sigma_k) \propto p(x_i|y_i, \mu_k, \sigma_k)p(y_i)p(\mu_k, \sigma_k)$. What if we want to do the same with the MLR model? How can we write the $posterior \propto likelihood \times prior$ since the likelihood is $p(y_i|x_i, w_k)$ and it already resembles the posterior in GMM?

Answer 1

Let me answer a few of the questions

2. Mixture models are typically weighted sum of individual probability density functions (or mass functions) from well known families such as Gaussian, Poisson, etc. So, as the individual components of MLR are not densities, it may not be appropriate to consider MLR as a mixture model.

3. Probability density functions are defined for continuous random variables (rv). Eq (1) is the probability mass function as you are looking at a discrete rv $Y_i$.

4. Answer to 3 should clarify.

5. For continuous rvs, PDF is not zero. It is probability that is zero. I kindly suggest you to revise the definitions of PDF, PMF and likelihoods as it can be a potential source of your confusion.

6. In GMMs, the probability of each class is determined using the posterior probability of the latent discrete random variable $Y$. It is not the likelihood. Because the rv is discrete, we pick the mode as the class of the observation $x$.