What makes the formula for fitting logistic regression models in Hastie et al "maximum likelihood"?

Question

I am learning logistic regression from The elements of statistical learning: data mining, inference, and prediction, by Trevor Hastie, Robert Tibshirani, Jerome H. Friedman.

Suppose $G$ is a random variable representing response taking class label in $\{1,\ldots,K \}$, and $X$ is a random vector representing predictor taking values in $\mathbb{R}^n$.

After modeling $P(G = k|X = x; \theta), k=1,\ldots,K, x \in \mathbb{R}^n$ by logistic model in section 4.4 Logistic Regression, the parameter $\theta$ is estimated from samples $\{(x_i,k_i), i=1,\ldots,N \}$ in section 4.4.1 Fitting Logistic Regression Models:

Logistic regression models are usually fit by maximum likelihood, using the conditional likelihood of $G$ given $X$. Since $P(G|X)$ completely specifies the conditional distribution, the multinomial distribution is appropriate. The log likelihood for $N$ observations is $$ℓ(θ) = \sum_{i=1}^N \log P(G = k_i|X = x_i; \theta) $$

My question is how the estimation method is maximum likelihood.

Generally speaking, MLE is to estimate parameter $\theta$ of a single distribution $p(z;\theta)$ from its i.i.d. samples $\{z_i, i=1,\ldots,N \}$ by maximizing $\log p(x_1,\ldots, x_N; \theta) = \sum_{i=1}^N \log p(x_i; \theta) $, the log of joint distribution of the samples.

The estimation for logistic regression as in the quote seems not using MLE, because:

1. $P(G = k|X = x_i; \theta), i=1,\ldots,N $ are $N$ different distributions, instead of just one distribution as for MLE;
2. in $ℓ(θ) = \sum_{i=1}^N \log P(G = k|X = x_i; \theta) = \log \prod_{i=1}^N P(G = k|X = x_i; \theta)$, I wonder how to explain that "${ (k_i|x_i), i=1,\ldots,N }$" are independent and therefore $\prod_{i=1}^N P(G = k_i|X = x_i; \theta)$ is their joint distribution, as I don't know how to understand $(k_i|x_i)$ which seems not make sense.

My guess answer is that $\{(x_i,k_i), i=1,\ldots,N \}$ are assumed to be i.i.d. samples of the single distribution for $(X,G)$. So MLE for the distribution of $(X,G)$ is to maximize $$ \log P((x_1,k_1),\ldots,(x_N,k_N)) = \log \prod_{i=1}^N P(x_i,k_i) $$ $$ = \log \prod_{i=1}^N P(k_i | x_i) P(x_i) = \log \prod_{i=1}^N P(k_i | x_i) + \log \prod_{i=1}^N P(x_i) $$ $P(k_i | x_i)$ is parameterized by $\theta$ as $P(k_i | x_i;\theta$, while $P(x_i)$ is assumed to be independent of $\theta$, so to solve for $\theta$ by MLE for the distribution of $(X,G)$ is equivalent to maximize $\log \prod_{i=1}^N P(k_i | x_i; \theta)$. I wonder if this is a reasonable explanation, and what yours are?

Thanks and regards!

Answer 1

Typically, $X_i$ are assumed to be fixed rather than random, so there is no real distribution $\prod_i P(X_i)$. This is certainly true in a fully controlled environment, where all variables can be manipulated by the researcher/designer/statistician. If you Google "logistic regression", most of the references will be coming from biostatistics, where there are two groups, such as presence and absence of a syndrom. (The model with more than two possible categories is more commonly known as a the multinomial logistic regression model, and is more widely used by economists studying the choice of a service or a good from a variety of options.) In both biomedical and economics examples, the data are observational, so most of the time there is only partial control for the covariates (such as the dose of a medicament). Factors like age, gender, etc. cannot be controlled for, and whether they should be treated as random or fixed is not a clear cut. However, if your interest is focused on the logistic regression parameters, then even assuming random covariates, your last term $\prod_i P(X_i)$ contains no information about $\theta$, so inference wrt $\theta$ can be conducted using only $P(k_i|X_i)$ part.

It's interesting that you never really specified the logistic regression: $$ P(G=k|X=x) = \frac{\exp( \beta_k'x )}{\sum_{j=1}^K \exp( \beta_j'x)},$$ with the normalizing constraint $\beta_1=0$, say. It is the logistic function used in modeling the binary outcome that gives the model the specific functional form and the name. There may be other forms of the dependence $P(G=k|X=x)$, such as probit, complementary log-log, etc., although multinomial logistic makes the best sense out of all of them, at least compared to the analysis of binary dependent variables where different functional forms kinda compete with one another. Assuming there are only two classes, $G\in\{0,1\}$, the probit model is $P(G=1|X=x) = \Phi(\beta'x)$, where $\Phi(z)$ is the standard normal cdf, and the cloglog model is $P(G=1|X=x)=1-\exp[-\exp(\beta'x)]$. Note that the latter corresponds to an asymmetric distribution (extreme value). Some readings on these other models may include J Scott Long's Regression Models for Limited Dependent Variables or Wooldridge's Econometric Analysis of Cross-Sectional and Panel Data. A lot of things in these books will appear odd to you as a computer scientist, though. But we must learn to respect our differences, don't we?