What is the difference between classical frequentist methods and likelihood methods?

Question

You may assume that I'm familiar with the material in Casella and Berger.

This question is identical to What is the difference between Fisherian vs frequentist statistics?; however, the question was closed for ambiguity and has no answers.

I speak of "likelihood methods" in context of the text In All Likelihood: Statistical Modelling and Inference Using Likelihood by Pawitan. Truth be told, I've only skimmed this text a few times.

Here is the description of the text provided in Amazon [emphasis added]:

Based on a course in the theory of statistics this text concentrates on what can be achieved using the likelihood/Fisherian method of taking account of uncertainty when studying a statistical problem. It takes the concept ot the likelihood as providing the best methods for unifying the demands of statistical modelling and the theory of inference. Every likelihood concept is illustrated by realistic examples, which are not compromised by computational problems. Examples range from a simile comparison of two accident rates, to complex studies that require generalised linear or semiparametric modelling.

The emphasis is that the likelihood is not simply a device to produce an estimate, but an important tool for modelling. The book generally takes an informal approach, where most important results are established using heuristic arguments and motivated with realistic examples. With the currently available computing power, examples are not contrived to allow a closed analytical solution, and the book can concentrate on the statistical aspects of the data modelling. In addition to classical likelihood theory, the book covers many modern topics such as generalized linear models and mixed models, non parametric smoothing, robustness, the EM algorithm and empirical likelihood.

Also, the review by "MathsEngineer" states [emphases added]:

As a practicing data analyst, I frequently find the standard "classical" statistical techniques either useless, cumbersome, or convoluted for all but the most straightforward applications. However, before running across the book, I thought my only other option would be to utilize Bayesian methods (which are infinitely more flexible and elegant) to perform inference with complex, non-normal/non-linear models. Unfortunately, solving Bayesian models relies on two rather unappealing concepts: a prior probability (so that all your calculations still yield probabilities),and Markov Chain Monte Carlo (a computationally intensive, sometimes non-convergent, algorithm that is only asymptotically correct wrt the true Bayesian posterior probability distribution.)

Reading Dr. Pawitan's book introduced me to a very satisfying "third way" as he calls it. Instead of force-fitting all uncertainty into a probability, the "likelihood" approach recognizes two types of uncertainty, which is both novel in statistics and extremely refreshing once you understand why two types are necessary. The first, which I would call "well calibrated" uncertainty, is analogous to a confidence interval for the mean of a normal sample. With this type, we know how often we would be wrong under repeated sampling from this population, so we have a good idea how well our method brackets the true mean i.e., well calibrated.

As I skim through the index, a lot of it looks like the statistics material in Casella and Berger (which has much more of a frequentist than Bayesian emphasis), as well as some linear models (with matrices). Thus, I don't see how this is so different from frequentist "classical" statistics.

I do understand the philosophical differences between frequentist and Bayesian statistics, but this "likelihood" approach I'm not as familiar with. Could someone elaborate on this?

Answer 1

Far from a complete answer (or a book):

The majority if not all frequentist techniques are various forms of method of moments estimators: $\hat{\beta}: 0 = \sum_{i=1}^n S_i(\hat{\beta})$ where the $S_i$ is simply some function you know will goes to zero when it estimates a thing you want. The $S$ is called an estimating function and solving it for a zero-sum is an estimating equation. Express the thing you want to estimate as a estimating function and just solve it. When the sample is IID, the estimating functions are independent RVs with non-singular variances, they have asymptotic properties that can be solved analytically or using the delta-method to find variances on the coefficients.

Some frequentist examples that are not ML: we often interpret the least squares coefficient as a slope. Why not just calculate all the pairwise slopes? Solve the estimating equation $0 = \hat{\beta}-\sum_{i=1}^n \sum_{j=1}^n (Y_i - Y_j)/(X_i - X_j) \mathcal{I}(i \ne j)$. Median unbiased estimators are frequentist but not maximum likelihood $0 = P(Y > X\hat{\beta}) - P(Y < X \hat{\beta})$. Many questions on CV itself deal with fitting logistic regression models to $Y$ observed as a proportion. There is nothing wrong with setting the $Y$ in the estimating equation for logistic regression: $0 = \mathbf{D}^T \mathbf{V}^{-1} \left( Y - g(\mathbf{X}\hat{\beta}\right))$ where $g$ is the inverse logit, $V_{ii} = g(X\beta)(1-g(X\beta))$ and $\mathbf{D}^T = \frac{\partial}{\partial \beta} g(\mathbf{X}\beta)$.

A bigger class of estimators called minimax estimators ultimately lead to estimating equations when the loss function is smooth. Minimax also has a lot of overlap with ML.

Whenever maximum likelihood is a regular exponential family, the score function (log likelihood) is an estimating equation. The root of the score equation maximizes the (log) likelihood: $0 = \sum_{i=1}^n S_i(\hat{\beta}) = \sum_{i=1}^n \frac{\partial}{\partial \beta} \log \mathcal{L}(\hat{\beta}; \mathbf{X}_{i,}, Y_i)$

However, many likelihoods do not easily lend themselves to estimating equations: models with missing data, models with measurement error, negative binomial models, mixture models, censored exponential survival outcomes, and Gaussian mixture models are all estimated by the EM algorithm or numerical integration. In those circumstances, the loss of information from the complexity of the data observation process was recovered by assuming a parametric model for the response.