# Finding ideal hypothetical data according to a regression model

Analogy

We have 10 athletes. Each athlete is represented as a binary feature vector. We make these athletes compete in a 100 metre race. We get real value numbers corresponding to the time they took in the race. We train a model from athlete features to time. Now, say we want to imagine a hypothetical athlete that runs the 100 metre race as quickly as possible, what feature vector (or vectors) would our model need to predict the fastest time?

As human beings it seems that we do this all the time. "Imagine how fast Usain Bolt would run if he had bionic legs." In fact, one could view any kind of training as the process of attempting to adjust our features towards those that we believe will make us perform better. How do we decide what those hypothetical features are?

Technical

Inverse regression as I understand it is the problem of learning the expectation of data given labels according to a model. It answers the question, if a model predicts a label, what observation was it that resulted in this prediction?

Say that we've learnt a Bayesian linear regression model from data that is a binary feature vector (e.g. $<0,1,1,0>$ indicating if a feature is active or not) and a response value $r \in \mathbb{R}$, how would I use inverse regression to tell me what hypothetical data would be required to give me a particular value of $r$. Or if not a particular value, the maximum/minimum value of $r$ that my model might predict given hypothetical data. I could of course just try all $2^n$ feature combinations where $n$ is the number of features. However, there must be a more efficient way of doing this?

• What kind of a model you have? If it's a linear regression, then crank up all variables to max/min depending on the sign, and you get your super athlete. Is there something you didn't mention in the question? – Aksakal Oct 13 '15 at 14:34
• Bayesian linear regression. By cranking up the variables do you mean choose high/low values of y and find x|y? I'm also not sure how to calculate the inverse prediction. – Michael Anslow Oct 13 '15 at 19:07
• No, I mean you put any arbitrary high number into a linear model and get high Y. I'm missing something – Aksakal Oct 13 '15 at 19:13
• I do not know what a high value of x means. That is why I want to find what 'high' value is with respect to y. For example, x might be $<\text{wears trainers}, \text{follows Atkins diet}>$ where we have a 1 if the statement is true and 0 if it is false. I might find that given an athlete $<1,0>$ I get a time of 8 seconds and given an athlete $<0,1>$ I get a time of 9 seconds. Clearly the value of x is only with respect to y and I want to find what my model would predict the highest value x would be with respect to y. We might not have seen x before so it might be $<1,1>$ or $<0,0>$. – Michael Anslow Oct 14 '15 at 10:11
• What I am looking for seems to be 'calibration' in machine learning/statistics. I'm still not sure how to apply it to a Bayesian linear regression model. – Michael Anslow Oct 14 '15 at 10:22

I have not been able to find a complete answer but I have found a solution to the more difficult part of the problem solved by either reverse prediction or inverse regression. The solution is more general than my question asked for as it allows for explanatory and dependent variables to be multivariate.

The solution described below are taken from a great thesis called "Dynamic Bayesian Approaches to the Statistical Calibration Problem" by Derick Lorenzo Rivers published in 2014.

There are a couple of caveats to this solution:

• This solution assumes absolute calibration: there is none or negligible error in the explanatory variables
• It is controlled calibration as the selection of the explanatory variables is not random

We are given observations $<\textbf{x}_i, \textbf{y}_i >$ where $\textbf{x}_i$ is a $p \times 1$ vector and $\textbf{y}_i$ is a $q \times 1$ vector. $\textbf{x}_i$ is an explanatory variable and $\textbf{y}_i$ is a dependent/observed variable. Our goal is to find an unknown explanatory vector $\mathbf{x}_0$ given an observation $\mathbf{y}_0$. A model for the standard regression from x to y is given by:

$\textbf{Y} = \textbf{1}\boldsymbol{\alpha}' + \textbf{XB}+ \textbf{E}$

where $\textbf{Y}$ and $\textbf{E}$ are ($n \times q$) matrices, $\textbf{X}$ is a ($n \times p$) matrix of fixed constants and $\textbf{1}$ is a ($n \times 1$) vector of ones. $\textbf{B}$ is a $p \times q$ matrix of unknown parameters, and $\boldsymbol{\alpha}$ is a $q \times 1$ vector of unknown parameters. We assume that $\textbf{X}$ is standardized having the average sum of squares equal to 1.

The classical estimator of $\hat{\textbf{B}}$ is $\hat{\textbf{B}} = (\textbf{X}'\textbf{X})^{-1}\textbf{X}'\textbf{Y}$ and $\hat{\boldsymbol{\alpha}}$ is $\hat{\boldsymbol{\alpha}} = \bar{\textbf{y}}$. Given these estimators the model for prediction of dependent variables given explanatory variables is given by:

$\textbf{y}_0 = \textbf{1}\boldsymbol{\alpha}' + \textbf{1}\textbf{x}'_0\textbf{B} + \textbf{E}^{\star}$,

where $\textbf{y}_0$ and $\textbf{E}^{\star}$ (the error/noise) are ($m \times q$) random matrices and $\textbf{x}_0$ is a $p \times 1$ vector of unknown values and $\textbf{1}$ is a ($m \times 1$) vector of ones. If $\mathbf{e}'_i$ is the $i^{th}$ row of $\mathbf{E}$, it is assumed that $E(\mathbf{e}_i) = 0, E(\mathbf{e}_i \mathbf{e}_i^T) = \pmb{\Gamma}$ and $\mathbf{e}_i \sim N(\mathbf{0}, \pmb{\Gamma})$ for $i = 1,2, \cdots, n$. If $\mathbf{e}_j^{\star'}$ is the $j^{th}$ row of $\textbf{E}^*$, $\mathbf{e}^{\star}_j$ satisfy the above also and it is assumed they are independent of the $\mathbf{e}'_i$.

## Reverse Prediction/Classical Calibration Estimation

The estimation of unknown explanatory variable $\mathbf{x}_0$ using reverse prediction is

$\hat{\textbf{x}}_0 = (\hat{\textbf{B}}\textbf{S}^{-1}\hat{\textbf{B}}')^{-1}\hat{\textbf{B}}\textbf{S}^{-1}(\textbf{y}_0-\bar{\textbf{y}})$

where $\textbf{S}$ is a ($q \times q$) matrix given by $\textbf{S} = \hat{\textbf{E}}'\hat{\textbf{E}} = (\textbf{Y}- \textbf{X}\hat{\textbf{B}})'(\textbf{Y}=\textbf{X}\hat{\textbf{B}})$ with $v = n-p=q$ degrees of freedom.

## Inverse Regression

An alternative to the classic calibration approach (\textit{inverse prediction}) is \textit{inverse regression} defined by the model,

$\hat{\textbf{X}} = \textbf{Y}\hat{\textbf{B}}_k$

and $(\mathbf{X}-\hat{\mathbf{X}})'(\mathbf{X}-\hat{\mathbf{X}})$ is minimized using the least squares method and the least squares estimate of $\mathbf{B}_k$ is

$\mathbf{B}_k = (\mathbf{Y}'\mathbf{Y})^{-1} \mathbf{Y}'\mathbf{X}$

For a given $1 \times p$ dimensional observation $\mathbf{y}_0$, $\hat{\mathbf{x}}_{0,k}$ is given by

$\hat{\mathbf{x}}_{0,k} = \mathbf{X}'\mathbf{Y}(\mathbf{Y}'\mathbf{Y})^{-1}\mathbf{y}'_0$

The argument for using reverse prediction or inverse regression for calibration has been a contentious issue that would require its own question.