3
$\begingroup$

Caveat lector: I am not sure what is meant by a "log-linear model". The Wikipedia page makes it seem as if log-linear model is an alternative term for exponential family.

This description of a book about graphical models says that graphical models are a subset of graphical models, i.e. that there exist log-linear models which are graphical models, but that every graphical model is a log-linear model.

Question: How and/or why should that be true?

Doesn't a graphical model refer to any statistical model for which the the conditional independence relationships between all of the random variables can be described by a graph?

If log-linear model refers to exponential families, then can't every exponential family be represented on a (possibly trivial) graph? And while exponential families are the most commonly studied types of models w.r.t. graphical models (e.g. see here or here), why couldn't the nodes of a graphical models correspond to random variables with, e.g., Laplace distributions?

In particular, it seems to me that it would make more sense if log-linear models were considered a subset of graphical models, not the other way around.

However, I am obviously wrong, so if you can explain to me why it would be greatly appreciated.

EDIT: I am now even more confused, because this review of the same book above says:

Graphical models have emerged in recent years as a unified framework for applied multivariate statistics encompassing such techniques as regression analysis, analysis of variance and covariance, log linear models, path analysis, factor analysis, and latent class models.

How can both be true? Was the description for the book just wrong?

$\endgroup$
2
$\begingroup$

EDIT 1: The following, seemingly contradictory, quotes, are both found in the preface of Joe Whittaker's book Graphical Models in Applied Multivariate Statistics.

... the development of log-linear models made it possible to formulate complex models for the dependencies between the variables cross-classifying a contingency table. The elucidation of the family of graphical models as a subset of log-linear models unravelled the connection between these models and the fundamental notion of conditional independence...

... graphical models span the set of all log-linear models...

In section 4.4.1.2 of Probabilistic Graphical Models: Principles and Techniques by Koller and Friedman, the term log-linear model appears to be defined (implicitly) as a method of parametrization of Markov random fields, being related explicitly to the Ising model from statistical physics and the notion of an energy function. A connection to exponential families seems implict as well in the treatment, but like a precise and explicit definition of log-linear model, is never given in the section. Wainwright and Jordan relate graphical models, exponential families, and the Ising model from statistical physics fairly explicitly, but never mention log-linear models. In both the preface and introduction to his book Graphical Models, Steffen Lauritzen indicates that a formal relationship between contingency tables (mentioned by Whittaker above as being related to log-linear models) and statistical physics was instrumental in the birth of the field of graphical models, although he never mentions the terms log-linear model or Ising model explicitly.

Based on the circumstantial evidence above, I am willing to make the following provisional conclusions, to be changed upon being presented with additional evidence:

(1) Researchers do not seem to often be able to agree on precise and unambiguous terminology.

(2) Exponential families are related to the Ising model of statistical physics (see also this question).

(3) There is a formal similarity between the Ising model in statistical physics (a special case of exponential family) and log-linear models used in the analysis of contingency tables.

(4) The study of this formal similarity led to the development of the notion of undirected graphical models, which are effectively a generalization of both the Ising model and log-linear models, and hence can often be parametrized using exponential families.

In particular, the converse claim that graphical models are a special case of log-linear models appears to be an unfortunate result of (1) researchers not speaking unambiguously about the topics they research, (2) a historical artifact that the impetus for developing the notion of undirected graphical model came from researchers hoping to explain at least a subset of log-linear models as being related to Ising models from statistical mechanics, and then discovering much more.


EDIT 2: On p. 200 Whittaker says that "...a concise treatment would place itself in the more general setting of exponential family models, and regard the log-linear and Gaussian models as special cases". This would imply that graphical models are more general. But on p. 198 he writes "The set of graphical models is located within the larger family of hierarchical log-linear models...". This seems contradictory, although the preceding paragraphs also seem to indicate that the author means only the special cases of Gaussian or multinomial distributions when referring to "graphical models".

Part of the issue also seems to be that the work is (by the standards of graphical models research) relatively dated, being from 1990. The author does cite, for example, Judea Pearl's paper from 1986 elucidating the difference between Bayesian networks and Markov random fields, but throughout the book never uses either of those two terms. The author uses both directed and undirected graphs, but I still have to look at the definitions more closely to see whether the conditional independence rules he uses those to define are the same as those implied by Bayesian networks and/or Markov random fields. I.e. the sense in which he uses "graphical model" could be much more restricted/limited than modern usage.

Wikipedia's definition of hierarchical log-linear model is not very useful in clarifying what they are and why they supposedly generalize graphical models. However, the next section seems helpful, inasmuch as it appears to use terminology the same way Whittaker does, thus clarifying his meaning somewhat:

A log-linear model is graphical if, whenever the model contains all two-factor terms generated by a higher-order interaction, the model also contains the higher-order interaction.

It also says that:

a graphical model can be represented by an undirected graph, where the vertices represent the variables and the edges represent the two-factor terms included in the model.

The above clearly seems to imply a more restrictive use of the term than modern usage. (E.g. not every distribution representable graphical model can be represented by a Markov random field (undirected graph), contradicting the above claim.)

The article also talks about decomposable models,

A log-linear model is decomposable if it is graphical and if the corresponding graph is chordal.

which sounds like the clique-tree construction for the sum-product algorithm. (I forgot exactly the if and only if statement in modern terminology for probability distributions represented by chordal graphs, but as far as I remember, the modern terminology is not "decomposable".)

The definition of graphical model given on Wikipedia sounds like the distinction (downwards closure) between simplicial complexes and more general hypergraphs. This makes sense inasmuch as graphical models are related to hypergraphs, see e.g. Lauritzen's book or this paper, by Robeva and Seigal. What Wikipedia refers to as "graphical models" seems to be referred to as "graphical models with maximal cliques" in that paper.

Graphical models defined by the maximal cliques of a graph correspond to hypergraphs in which we introduce a hyperedge for each maximal clique.

... Graphical models are sometimes required to factorize according to the maximal cliques of a graph. We see later how our set-up specializes to this case. Models with cliques that are not necessarily maximal can be called hierarchical models... [emphasis mine]

Also the definition the authors of that paper are using appears to be a more general factor graph definition, rather than the more specialized Bayesian network or MRF definitions. In Section 1.2. of Frey, Graphical Models for Machine Learning and Digital Communication, it is explained how factor graphs generalize both Bayesian networks and Markov random fields (MRFs).


EDIT 3: Reading the definition of "undirected independence graph" given in Chapter 3 in Whittaker's book, and comparing them with those in Chapter 4 of Friedman, Koller, it seems that "undirected independence graph" in Whittaker's terminology corresponds to "Markov Random Field" in more modern terminology. (Also he talks about "directed independence graphs" which today would be called Bayesian networks, at least if I understand it correctly.)

When Whittaker refers to "graphical models" in Chapter 7, talking about log-linear models, he appears to mean it as a shorthand for "graphical log-linear models", which he gives a specific definition, particular only for multinomial distributions (pp. 207-208):

Given an [undirected] independence graph G [Markov random field], the cross-classified Multinomial distribution for the random vector $X$ is a graphical model for $X$ if the distribution of $X$ is arbitrary apart from constraints of the form that for all pairs of coordinates not in the edge set $E$ of $G$, the $u$-terms containing the selected coordinates are identically zero. More explicitly, the density of a Multinomial graphical model is $$\log f_K(x) = \sum\limits_{a \subseteq K} u_a(x_a) $$ subject to the constraints that $u_a = 0$ if $\{i,j\} \subseteq a$ and $(i,j)$ is not in the edge set $E$. The parameters of the graphical model are the remaining $u$-terms that are not set to zero.

He also calls this later on p. 208, example 7.3.2. a purely graphical model. (Only undirected graphs appear in Chapter 7, so that is why I feel justified in assuming this is what Whittaker meant here.)

We can see that this is strictly less general than Markov random fields because of the distributional assumptions. However, one can even deduce this from Whittaker's book -- in Example 7.3.2. he gives an example of a "non-graphical-model" which has the same "[undirected] independence graph". Also in Example 7.3.3. he gives an example of a "hierarchical", "but ... not graphical" model along with a picture of its independence graph! So clearly the naive intuition for what one would think "graphical model" means does not match Whittaker's definition. He also gives an explicit "independence graph" for a non-hierarchical log-linear model, so clearly Markov random fields generalize not only "graphical models" (in Whittaker's meaning) but also hierarchical log-linear models, such that my original supposition, that Markov random fields actually generalize log-linear models, appears correct. The following quote from Whittaker, p. 209 supports this even more so (if one translates "independence graph" to "undirected independence graph" to "Markov random field"):

... note that every distribution with a log-linear expansion [emphasis mine] has an interaction graph constructed by connecting any pair of vertices $(i,j)$ by an edge if there is an interaction term $u_a$ in the expansion in which the set $a$ contains both $i$ and $j$. Application of the factorization criterion makes it evident that the interaction graph is an independence graph [the emphasis here is again mine].

The difficulty in these definitions seems to be related to how a Markov random field doesn't necessarily specify all higher-order interactions uniquely, in the sense that the same Markov random field can correspond to distinct factor graphs.

In particular, the above-stated phenomenon appears to be related to Proposition 7.3.1. on p. 209 of Whittaker (A hierarchical log-linear model is graphical if and only if its maximal $u$-terms correspond to the cliques in the independence graph), the statement about the definition of hierarchical in Robeva, Seigal's paper, as well as Figure 4.8 on p. 123 of Koller and Friedman's book, which gives an example of the same Markov Random Field which corresponds to two distinct factor graphs (presumably Figure 4.8.a is a "graphical model" in Whittaker's parlance, but not Figure 4.8.b). (See also the comment at the beginning of p. 117 of Friedman/Koller: "In other words, it is not the case that every pair of nodes $X$ and $Y$ that are not separated in $\mathcal{H}$ (the graph) are dependent in every distribution $P$ which factorizes over $\mathcal{H}$".)

Whittaker's definition of "graphical model" in Chapter 7 (about log-linear models) isn't that bad inasmuch as it only is intended to require that the MRF/undirected independence graph is somehow a "good" depiction of the conditional independence structure, and for that reason could be excused as simply being a different choice of priorities in terminology between different time periods. However, his definition isn't self-consistent, at least to the extent that it conflicts with the definition of "graphical model" he gives on p. 156 (in Chapter 6), which seems to more closely compare with modern usage:

Consider a $k$-dimensional random vector $X=(X_1, X_2, \dots, X_k)$ and an independence graph $G=(K,E)$. A graphical model for $X$ is a single family of probability distributions for $X$, that satisfy the pairwise conditional independence restrictions inherent in $G$, but are otherwise arbitrary [emphasis mine].

It might be the case that I am mis-interpreting the definition (indeed, this is actually the most likely possibility), but in case my interpretation is correct, using this definition of "graphical model" in Chapter 6, every log-linear model in Chapter 7 would be a "graphical model", not just the "graphical models" (i.e. the "perfect graphical models" or "the definition of graphical models for log-linear models/multinomial distributions").

As for the relationship of log-linear models with exponential families, it also appears to be the case (although I could still be mistaken), that, indeed, log-linear models are a special case. In particular, it seems to me, from my cursory glance at Chapter 7 of Whittaker, as well as this excellently written introduction (see section 3.2. starting on page 13) by Larry Wasserman, that log-linear model is just a fancy term for the natural/canonical parametrization of the multinomial distribution(s) as an exponential family. To the extent that discrete graphical models in the modern sense, sensu lato, encompass all multinomials and more, it seems clear that they do generalize log-linear models, and not vice versa, and that any statements to the contrary are the by-product of dated terminology/alternative specialist definitions. (The $\beta$'s in Professor Wasserman's notes seem to correspond to Whittaker's $u$'s. Also, the fact that log-linear models appear as a sub-section of Wasserman's discussion of undirected graphical models (in the modern sense), as well as a sub-section of Friedman/Koller's, also suggests the same/original conclusion).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.