Regression as mutual information minimization

Question

I am trying to see if mutual information can be used as an objective function in a generalized formulation of the linear regression without the normal distribution assumption for the residual error.

Suppose $$y=ax+z$$ where $a$ is a constant, $y,x$ and $z$ are random variables. The conventional linear regression formulation assumes $z$ is distributed normally conditioned on $x$ with $\mathbf E[z|x]=0$. Dropping this normality assumption, we now let $z$ take on an arbitrary distribution. There is no outlier in the observation. I formulate the linear problem regression problem to estimate $a$ as $$\min_a I(y-ax;x),$$ for observed $(x,y)$ and unknown $z$. $I(u;v)$ denotes the mutual information between random variables $u$ and $v$. When $(x,z)$ are specialized to be jointly normal, the minimization problem becomes $$\min_a \text{cor}(y-ax,x)$$ where cor$(u,v)$ stands for the correlation between random variables $u$ and $v$. Of course, given finite sampling of continuous random variables $x,y$, it begs the question of how I would estimate the probability distribution of $(x, y)$.

Is this a meaningful approach? Has there been any research in this regard?

Answer 1

I don't completely understand your notation; my apologies if I misunderstand. But I think MI is too weak for regression, because it only enforces dependence between variables, rather than any specific relationship (as regression requires). (In fact, it is this generality that makes MI better for detecting hidden statistical relationships.)

For example, in some latent variable generative models (e.g., VAEs, see ELBO surgery: yet another way to carve up the variational evidence lower bound, Hoffman & Johnson), we maximize the mutual information between the data $x$ and its latent representation $z$. But we have to constrain this with other terms (reconstruction likelihood and matching the prior). Indeed, optimizing mutual information is a very active research area in modern machine learning (which I suppose could be considered research in the area of regression), especially in models like variational auto-encoders and generative adversarial networks (see the bottom of this answer).

Separately, something interesting to you might be Bayesian linear regression, which looks at regression from a more general probabilistic viewpoint.

---

On the other hand, it did make me think of the following simple idea, since mutual information is closely related to KL-divergence: $$ \mathcal{I}[A;B] = \mathbb{E}_{B}\left[ \mathfrak{D}_\text{KL}[P(A|B)\,||\,P(A)] \right] $$

Suppose we assume the following data generation process: $$ x\sim\mathcal{U},\;\; y \,|\, x\sim\mathcal{N}(\alpha_r x +\beta_r, \sigma^2_r) $$ We want to learn the parameters $\alpha,\beta$ of a stochastic regressor: given $x$, $$ \hat{y}\sim\mathcal{N}(\alpha x + \beta, \sigma^2) $$ noting that the KL-divergence is asymmetric, what is the KL-divergence on the conditional distributions between the data generator and the regressor? \begin{align} \mathfrak{D}_\text{KL}[p(y|x)\,||\,p(\hat{y}|x)] &= \log(\sigma_\hat{y} / \sigma_y) + \frac{\sigma_y^2 + (\mu_y - \mu_{\hat{y}})^2}{2\sigma_{\hat{y}}^2} - \frac{1}{2}\\ &= \log(\sigma) - \log(\sigma_r) + \frac{\sigma_r^2 + (\alpha_r x +\beta_r - \alpha x - \beta)^2}{2\sigma^2} - \frac{1}{2}\\ \end{align} Suppose we have data $D=\{ (x_i,y_i) \}$. Then we want to minimize \begin{align} \sum_i \mathfrak{D}_\text{KL}[p(y_i|x_i)\,||\,p(\hat{y}_i|x_i)] &= c + \frac{1}{2\sigma^2} \sum_i (y_i - \alpha x_i - \beta)^2 \end{align} wrt $\alpha$ and $\beta$, where $c$ is a constant (assuming $\sigma$ and $\sigma_r$ are). But this just amounts to minimizing the squared error loss here.

Answer 2

You're proposing to minimize the mutual information between the residuals and the inputs. Minimal information between the residuals and inputs is indeed a desirable property. But, it's not generally sufficient for regression because it doesn't force the residuals to be small. For example, note that any constant value can be added to the model--thereby making the residuals arbitrarily large and the fit arbitrarily poor--without affecting the mutual information.