5
$\begingroup$

Is there a well-known relationship between least-squares regression and information theory? I've just started reading about information theory. It seems almost trivial to say that the regression coefficients from least-squares tell us how much information the predictor variables hold about the dependent variable, but I don't know if this has been studied in any depth. For a simple regression with a single predictor variable, I arrive at the following. Does it make any sense? Does it extend to multiple linear regression, general linear models etc?

Information theory

The information entropy of a random variable $X$ is given by

$$H(X) = \int \mathrm{P}(x)\,\mathrm{I}(x) dx = -\int {\mathrm{P}(x) \ln \mathrm{P}(x)} ~dx,$$ where $\mathrm{P}(x)$ is the probability density function, and $I(x) = -\text{ln}(\mathrm{P}(x))$ is the self information of $x$.

Least-squares

Now suppose we have a dependent variable, $y$, and a predictor variable, $p$. Using least-squares regression We can write a simple linear model

$$y = \beta_{yp} p + e.$$ It seems to me that the regression coefficient $\beta_{yp}$ is a measure of the information held by $p$ about $y$. If we re-write the model as

$$y = -\text{ln}(e^{-\beta_{yp}}) p + e,$$ where $\beta_{yp} = -\text{ln}(e^{-\beta_{yp}})$ then we have the regression coefficient in a similar form to self information. If we take the absolute value of the beta coefficicient, $\mathopen|\beta_{yp}\mathclose|$, then the quantity $e^{-\mathopen|\beta_{yp}\mathclose|}$ looks like the probability density function $P(\mathopen|\beta_{yp}\mathclose|)$ over all possible $\beta$ coefficients. It relates to how likely it is that we got a beta coefficient with a magnitude as large as this. In turn, we find the information entropy to be

$$H(B) = - \int_{0}^{\infty} e^{-\mathopen|\beta_{yp}\mathclose|} \ln(e^{-\mathopen|\beta_{yp}\mathclose|}) ~d\mathopen|\beta_{yp}\mathclose|= e^{-\mathopen|\beta_{yp}\mathclose|}(\mathopen|\beta_{yp}\mathclose| - 1) \bigg|_{0}^{\infty}= 1.$$

Hope that makes some sense. Let me know what you think, if you have any references or suggestions. Thanks.

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ One obvious problem is that $\beta_{yp}$ can be negative in general. $\endgroup$ – mpiktas Sep 11 '15 at 10:47
  • $\begingroup$ Indeed, I was thinking we could simply use $\mathopen| \beta_{yp} \mathclose|$ though. That makes sense, because whether the slope is positive or negative, it still gives information about the dependent variable. $\endgroup$ – Michael Andrew Bentley Sep 11 '15 at 10:56
  • $\begingroup$ Actually it would only need to be $\mathopen| \mathrm{Cov}(y,p) \mathopen|$ because $\mathrm{Var}(p)$ is positive by definition. $\endgroup$ – Michael Andrew Bentley Sep 11 '15 at 10:57
  • 1
    $\begingroup$ On the second look, the first integral is the over the real line, only this is omitted, i.e. this is definite integral and $H(X)$ is a number. Your integral $H(B)$ seems to be indefinite. If that, it contains an error, since $\int e^tln(e^t)dt=\int te^tdt=e^t(t-1)+C$. $\endgroup$ – mpiktas Sep 11 '15 at 11:04
  • $\begingroup$ Yes well spotted! I've amended the question. $\endgroup$ – Michael Andrew Bentley Sep 11 '15 at 11:58
6
+50
$\begingroup$

The entropy of the random variable $Y$, $H(Y$) is sometimes called a measure of your "uncertainty" about $Y$. What if you know about this other variable, $X$? Your uncertainty about $Y$ given $X$ will go down. This reduction is called mutual information and is written $I(Y;X) = H(Y) - H(Y|X)$. From an information point of view, I think this is what you want to know: how is the information that $X$ gives me about $Y$ reflected in the regression coefficient? I'll show how to estimate the mutual information in terms of the regression coefficient, under some assumptions.

One way to approach this problem is to assume that the variables are normally distributed. Suppose $X$ is a random variable drawn from a Gaussian with mean $\mu_x$ and standard deviation $\sigma_x$. If $y = \beta_{y,x} x + \epsilon$ and epsilon is Gaussian noise (uncorrelated with $x$) with standard deviation $\sigma_x$, then the Pearson correlation coefficient, $\rho^2 = \beta_{y,x}^2 / (\beta_{y,x}^2 + \sigma_\epsilon^2/\sigma_x^2)$. Under these assumptions, the correlation coefficient is related to the mutual information: $$I(Y;X) = -1/2 \log(1-\rho^2) = 1/2 \log\left(1 + \beta_{y,x}^2 / (\sigma_\epsilon^2/\sigma_x^2)\right) $$ (using the definition of entropy for normal distributions). Actually, this is even true under some slightly weaker assumptions. If you really just want the entropy of $Y$, you can write it as a function of this expression as $H(Y) = I(X;Y) + H(Y|X)$. This type of analysis comes up in analyzing the capacity of the additive white Gaussian noise channel.

Just a few qualitative comments about this solution. This mutual information is always non-negative and zero if $\beta=0$. I guess you can interpret $\sigma_x/\sigma_\epsilon$ as a signal-to-noise ratio (SNR). If the SNR is small then it washes out any informative value that $x$ has about $y$.

Generally, the regression coefficient has no relationship to mutual information. I.e., you can find distributions with fixed $\beta$ where MI can be anything. Heuristically, though, it makes sense to think of the equation above as a lower bound.

For another connection between regression and information theory, I can also suggest this paper showing the relationship between linear regression and transfer entropy.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I'm not sure I do want mutual information rather than self information, although I confess I'm a little confused about the concept of the latter. The context I'm interested in is that I have a set of $y$'s that I have measured (traits of an organism, e.g. height), and I have a set of $p$'s that I know (presence or absence of a gene). I want to make a statement about whether my genes hold information about a particular trait. It seems intuitive to me that if $b_{yp}$ is large, or rather $\mathrm{Cov}(y,p)$ is large, I can say that the genes hold (self?) information about the trait. $\endgroup$ – Michael Andrew Bentley Sep 12 '15 at 9:48
  • 1
    $\begingroup$ This sounds exactly like a scenario where you want mutual information. The entropy or "self-information" of a trait will not depend on the regression coefficient. It's just, roughly, a measure of how much that trait varies. P.S. you might want to check out this paper about estimating covariance matrices in this context. P.P.S. Your traits are typically continuous but genes are discrete. Typically different estimators are used in that context, I can give pointers if you are interested. $\endgroup$ – Greg Ver Steeg Sep 13 '15 at 17:27
  • 1
    $\begingroup$ I would say that you definitely want mutual information, but not link to regression coefficients. Basically mutual information is a non linear generalisation of correlation. The mutual information between y and any invertible transformation of x is unchanged. $\endgroup$ – seanv507 Sep 13 '15 at 19:22
  • $\begingroup$ @GregVerSteeg, thanks. I've done a little more reading and I'm on board with the mutual information point at last! Pointers on the discrete --> continuous estimators would be very helpful yes please Greg. $\endgroup$ – Michael Andrew Bentley Sep 14 '15 at 9:57
  • $\begingroup$ @seanv507 The reason I wanted to link information to regression coefficients is that these are widely used in the fields of quantitative genetics. In my work I emphasise that thinking in terms of information is more general than thinking in terms of genes (which are just one way in which information is stored). I therefore need to relate my work to existing, mostly successful, approaches! $\endgroup$ – Michael Andrew Bentley Sep 14 '15 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.