# Regression as mutual information minimization

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error.

Suppose $$y=ax+z$$ where $$a$$ is a constant, $$y,x$$ and $$z$$ are random variables. The conventional linear regression formulation assumes $$z$$ is distributed normally conditioned on $$x$$ with $$\mathbf E[z|x]=0$$. Dropping this normality assumption, we now let $$z$$ take on an arbitrary distribution. There is no outlier in the observation. I formulate the linear problem regression problem to estimate $$a$$ as $$\min_a \big(I(y-ax;x)\big),$$ for observed $$(x,y)$$ and unknown $$z$$. $$I(u;v)$$ denotes the mutual information between random variables $$u$$ and $$v$$. When $$(x,z)$$ are specialized to be jointly normal, the minimization problem becomes $$\min_a \big(\text{cor}(y-ax,x)\big)$$ where cor$$(u,v)$$ stands for the correlation between random variables $$u$$ and $$v$$. Of course, given finite sampling of continuous random variables $$x,y$$, it begs the question of how I would estimate the probability distribution of $$(x, y)$$.

Is this a meaningful approach? Has there been any research in this regard?

I don't completely understand your notation; my apologies if I misunderstand. But I think MI is too weak for regression, because it only enforces dependence between variables, rather than any specific relationship (as regression requires). (In fact, it is this generality that makes MI better for detecting hidden statistical relationships.)

For example, in some latent variable generative models (e.g., VAEs, see ELBO surgery: yet another way to carve up the variational evidence lower bound, Hoffman & Johnson), we maximize the mutual information between the data $$x$$ and its latent representation $$z$$. But we have to constrain this with other terms (reconstruction likelihood and matching the prior). Indeed, optimizing mutual information is a very active research area in modern machine learning (which I suppose could be considered research in the area of regression), especially in models like variational auto-encoders and generative adversarial networks (see the bottom of this answer).

Separately, something interesting to you might be Bayesian linear regression, which looks at regression from a more general probabilistic viewpoint.

On the other hand, it did make me think of the following simple idea, since mutual information is closely related to KL-divergence: $$\mathcal{I}[A;B] = \mathbb{E}_{B}\left[ \mathfrak{D}_\text{KL}[P(A|B)\,||\,P(A)] \right]$$

Suppose we assume the following data generation process: $$x\sim\mathcal{U},\;\; y \,|\, x\sim\mathcal{N}(\alpha_r x +\beta_r, \sigma^2_r)$$ We want to learn the parameters $$\alpha,\beta$$ of a stochastic regressor: given $$x$$, $$\hat{y}\sim\mathcal{N}(\alpha x + \beta, \sigma^2)$$ noting that the KL-divergence is asymmetric, what is the KL-divergence on the conditional distributions between the data generator and the regressor? \begin{align} \mathfrak{D}_\text{KL}[p(y|x)\,||\,p(\hat{y}|x)] &= \log(\sigma_\hat{y} / \sigma_y) + \frac{\sigma_y^2 + (\mu_y - \mu_{\hat{y}})^2}{2\sigma_{\hat{y}}^2} - \frac{1}{2}\\ &= \log(\sigma) - \log(\sigma_r) + \frac{\sigma_r^2 + (\alpha_r x +\beta_r - \alpha x - \beta)^2}{2\sigma^2} - \frac{1}{2}\\ \end{align} Suppose we have data $$D=\{ (x_i,y_i) \}$$. Then we want to minimize \begin{align} \sum_i \mathfrak{D}_\text{KL}[p(y_i|x_i)\,||\,p(\hat{y}_i|x_i)] &= c + \frac{1}{2\sigma^2} \sum_i (y_i - \alpha x_i - \beta)^2 \end{align} wrt $$\alpha$$ and $$\beta$$, where $$c$$ is a constant (assuming $$\sigma$$ and $$\sigma_r$$ are). But this just amounts to minimizing the squared error loss here.

• Very interesting and informative answer. However, there may indeed be a misunderstanding. I have dropped the normality of the residual $z$ and want to explore the linear regression in that non-Gaussian setting. All the alternative methods you have listed assume normality. – Hans Jan 12 at 19:00
• @Hans If you're interested in changing assumptions like that, maybe look into robust regression? For instance, when outliers are more prevalent, a $t$-distribution can be a better assumption than normality. (The computer vision book by Prince discusses this a bit.) Or a mixture model. Optimization with robust stats is an old and still widely used/researched/applied area I think. – user3658307 Jan 12 at 22:46
• I assume there is no outlier here. More importantly, I do not understand your objection to my MI formulation of the regression. "MI is too weak for regression, because it only enforces dependence between variables, rather than any specific relationship (as regression requires)". I have specified the relation $y=ax+z$ with $(x,y)$ observed and $z$ unknown. I am trying to make $z=y-ax$ as orthogonal to $x$ as possible in the metric of MI. This reproduces the least square result when $(x,y)$ is normal. Why is this not good? – Hans Jan 14 at 5:46
• Consider your formulation using the K-L divergence. Your K-L divergence measures the distance between the data generation process probability and the regressor probability. I chose not to use K-L precisely because we do not know the probability of data generation process and measuring the distance would be ill-defined. I think you get OLS result only because the variance is a constant. I doubt you will obtain a good result for heteroscedastic and especially for more general unknown data generation process. – Hans Jan 14 at 6:21
• @Hans My objection was the same as the other answerer. I'm not saying you are wrong; I would have to look at it a bit more, especially with your edits. In the mean time, maybe look at these: math.stackexchange.com/questions/1926665/…, stats.stackexchange.com/questions/171916/…, which seem to be relevant. – user3658307 Jan 14 at 12:59

You're proposing to minimize the mutual information between the residuals and the inputs. Minimal information between the residuals and inputs is indeed a desirable property. But, it's not generally sufficient for regression because it doesn't force the residuals to be small. For example, note that any constant value can be added to the model--thereby making the residuals arbitrarily large and the fit arbitrarily poor--without affecting the mutual information.

• While it is true that the mutual information is invariant under any injective transformation (not just adding any constant), it is irrelevant to this formulation since the only variable here is the linear coefficient $a$ and there is no extra degree of freedom. The constant $b$ is simply computed after $a$ is obtained as $b=\mathbf E[z]=\mathbf E[y]-a\mathbf E[x]$. For example, when $(x,z)$ are jointly normal, you obtain the same unique $a$ and $b$ as in the least square method. – Hans Jan 12 at 20:28