Is there a well-known relationship between least-squares regression and information theory? I've just started reading about information theory. It seems almost trivial to say that the regression coefficients from least-squares tell us how much information the predictor variables hold about the dependent variable, but I don't know if this has been studied in any depth. For a simple regression with a single predictor variable, I arrive at the following. Does it make any sense? Does it extend to multiple linear regression, general linear models etc?
Information theory
The information entropy of a random variable $X$ is given by
$$H(X) = \int \mathrm{P}(x)\,\mathrm{I}(x) dx = -\int {\mathrm{P}(x) \ln \mathrm{P}(x)} ~dx,$$ where $\mathrm{P}(x)$ is the probability density function, and $I(x) = -\text{ln}(\mathrm{P}(x))$ is the self information of $x$.
Least-squares
Now suppose we have a dependent variable, $y$, and a predictor variable, $p$. Using least-squares regression We can write a simple linear model
$$y = \beta_{yp} p + e.$$ It seems to me that the regression coefficient $\beta_{yp}$ is a measure of the information held by $p$ about $y$. If we re-write the model as
$$y = -\text{ln}(e^{-\beta_{yp}}) p + e,$$ where $\beta_{yp} = -\text{ln}(e^{-\beta_{yp}})$ then we have the regression coefficient in a similar form to self information. If we take the absolute value of the beta coefficicient, $\mathopen|\beta_{yp}\mathclose|$, then the quantity $e^{-\mathopen|\beta_{yp}\mathclose|}$ looks like the probability density function $P(\mathopen|\beta_{yp}\mathclose|)$ over all possible $\beta$ coefficients. It relates to how likely it is that we got a beta coefficient with a magnitude as large as this. In turn, we find the information entropy to be
$$H(B) = - \int_{0}^{\infty} e^{-\mathopen|\beta_{yp}\mathclose|} \ln(e^{-\mathopen|\beta_{yp}\mathclose|}) ~d\mathopen|\beta_{yp}\mathclose|= e^{-\mathopen|\beta_{yp}\mathclose|}(\mathopen|\beta_{yp}\mathclose| - 1) \bigg|_{0}^{\infty}= 1.$$
Hope that makes some sense. Let me know what you think, if you have any references or suggestions. Thanks.