distance between regression models Consider two multivariate linear regression models (vector inputs and outputs) with the same domain observations.  Namely, let:

*

*$X \in \mathcal{R}^{a \times N}$ be a matrix of domain observations (design matrix)

*$Y_i \in \mathcal{R}^{b \times N_i}$ be one (of two) co-domain observations matrices

where corresponding columns of $X$ and $Y_i$ refer to paired vector observations.  One can find the matrix, $\beta_i$, which maps each observation in $X$ to $Y_i$ as $y = \beta x$ while minimizing the mean squared error as $\beta_i = Y_i X^+$ where $X^+$ is the right pseudo-inverse of $X$.
Is there a natural metric between these two regression models?
I'm looking for something which captures the covariances in $Y_i$ (so metrics on $\beta$ directly, the space of mappings from $X$ to $Y$, don't fit the bill since $\beta_0 = \beta_1$ does not imply $Y_0 = Y_1$.)
 A: A model is really just a distribution;$^\dagger$ you can measure the distances between $Y_0$ and $Y_1$ (or rather, their distributions) via metrics like statistical distances. For example, f-divergences (which encompass the total variation distance, squared Hellinger distance, and Kullback-Leibler divergence).
$^\dagger$By that I mean $Y_{0, i} | X_i \sim \text{Normal}_i$ for $i \in 1, 2, \dots, n$ where $n$ is the number of observations you have. Same goes for $Y_1, i | X_i$.
References:

*

*Example on how to compute the Hellinger distance for Gaussian distributions

*Quick tutorial on Monte Carlo integration (use it to approximate expectations, e.g., in this case, f-divergences.)

A: The F-stat contains a ratio which describes the increased error in going from a full model to the reduced model ("reduced" because it has a subset of explanatory $x$ variables so its average residuals are not smaller).  In a similar spirit, we could measure model similarity by examining the increase in error in swapping the betas (i.e. applying $\beta_0$ to $Y_1$ and $\beta_1$ to $Y_0$).
Let the residual matrix of data i under model j be:
$\mathcal{E}_{i|j} = \beta_j X - Y_i$
we can compute the mean squared error of these as:
$MSE_{i|j} = \frac{1}{N_i} Tr (\mathcal{E}_{i|j}\mathcal{E}_{i|j}^T)$
so that the "swapped increase in mse" is:
$\frac{MSE' - MSE}{MSE}$
where $MSE' = MSE_{0|1} + MSE_{1|0}$ is the swapped mean square error and $MSE = MSE_{0|0} + MSE_{1|1}$ is the original mean square error.
