# Orthogonal parametrization

In general inference, why orthogonal parameters are useful, and why is it worth trying to find a new parametrization that makes the parameters orthogonal ?

I have seen some textbook examples, not so many, and would be interested in more concrete examples and/or motivation.

-
Orthogonal parameterization has a broad range of uses. Please be more specific. What are you applying it to? –  John Jul 31 '10 at 15:48

In Maximum Likelihood, the term orthogonal parameters is used when you can achieve a clean factorization of a multi-parameter likelihood function. Say your data have two parameters $\theta$ and $\lambda$. If you can rewrite the joint likelihood:

$L(\theta, \lambda) = L_{1}(\theta) L_{2}(\lambda)$

then we call $\theta$ and $\lambda$ orthogonal parameters. The obvious case is when you have independence, but this is not necessary for the definition as long as factorization can be achieved. Orthogonal parameters are desirable because, if $\theta$ is of interest, then you can perform inference using $L_{1}$.

When we don't have orthogonal parameters, we try to find factorizations like

$L(\theta, \lambda) = L_{1}(\theta) L_{2}(\theta, \lambda)$

and perform inference using $L_1$. In this case, we must argue that the information loss due to excluding $L_{2}$ is low. This leads to the concept of marginal likelihood.

-
I like your explanation (+1)-- it is probably more to the point for Alekk. I'll leave mine up in the hope that it casts a different, perhaps useful, light on the role of orthogonality. –  John L. Taylor Jul 31 '10 at 20:41
Can you comment on how your definition of marginal likelihood relates to the one given in the wiki link? (See: en.wikipedia.org/wiki/Marginal_likelihood) –  user28 Aug 1 '10 at 1:57
All the cites on that page (Bos; MacKay/p.29) are to Bayesian concepts of likelihood (post ~ lik x prior). There's a connection, but it's not the marginal likelihood in the sense of Fisher and Likelihood theory. [see comment on Berger paper here: stats.stackexchange.com/questions/1045/… ] –  ars Aug 1 '10 at 4:37

This is a good, if underspecified question.

Simply put, obtaining an orthogonal parametrization allows for parameters of interest to be conveniently related to other parameters, particularly in establishing needed minimizations. Whether or not this is useful depends on what you are trying to do (in the case of some physics problems, for instance, orthogonal parametrization may obscure the symmetries of interest).

In the case of statistical inference, orthogonal parametrization can allow the use of statistics by way of minimization (or its dual) on orthogonal parameters. For instance, Cox and Reid use the orthogonality of nuisance parameters (and their appropriately applied maximum likelihood estimates) to construct a generalization of a liklihood ratio statistic for a parameter of interest.

To see how orthogonality allows for this requires an understanding of the properties of commonly used mathematical spaces and the construction of estimators, which is essentially an issue of information geometry. See Information Geometry, Bayesian Inference, Ideal Estimates and Error Decomposition for a lucid, but technical description of orthogonality and its role in statistical inference.

-
[Ack, deleted a previous comment when I thought it would edit, back to try again.] This is interesting though I don't understand information geometry +1. I'm intrigued by this question at quora, which is unfortunately unanswered: quora.com/… –  ars Aug 1 '10 at 17:59
I had wondered where that went! If I get some time, I think I'll try to answer that question over at quora, though I should admit that I an information geometry novice. Most of what I know of it is incidental to an an effort to pin down how the features of mathematical spaces (topological, algebraic, and otherwise) enable the application of learning methods. This, in turn, is part of a quite difficult pet project of mine. –  John L. Taylor Aug 1 '10 at 18:51
If you would like to correspond about any of these subjects feel free to email me at johnnylogic at gmail. –  John L. Taylor Aug 1 '10 at 19:09
Cool, thanks. :) –  ars Aug 2 '10 at 23:49