4
$\begingroup$

I want to build a linear model to predict a scalar output from a vector of noisy scalar variable measurements.

I have two separate training data sets. One has output data and corresponding exact variable measurements. The other has exact variable measurements and corresponding noisy variable measurements. The noisy measurements of some variables are noisier (higher error variance) than others, and the noisy measurements of some variables are biased. I do not have a single data set with output data and corresponding noisy variable measurements.

How should I build my linear model? Should I use the exact variable measurements and ignore the fact that when the model is applied/used, noisy measurements of the variables will be input to the model? Or is there some way I can make use of what I can figure out about the noisy measurements of each variable when building the model? Can R help me with this problem?

| cite | improve this question | |
$\endgroup$
  • 2
    $\begingroup$ Suppose the exact variable values are $x_i$, the exact output value is $y$, observed variable values are $\hat{x}_i$ and the observed output value is $\hat{y}$. Is it true that in your model $y = \sum_i w_i x_i$ (i.e. the exact output is a linear function of the exact variable values)? Then, if you had access to $x_i$ and $\hat{y}$, this would be standard regression. But the issue is that you don't have $x_i$, but only $\hat{x}_i$, so you want to build a model of the form $\hat{y} = \sum w_i \hat{x}_i$, even though the true model is $y = \sum_i w_i x_i$? $\endgroup$ – SheldonCooper Mar 17 '11 at 0:17
  • $\begingroup$ Thanks @SheldonCooper - your notation helps. I think you have almost everything right. I have two data sets $D_1$ and $D_2$. $D_1$ contains a vector of observed output variables $Y_1$ and a matrix with corresponding exact variable values $X_1$. $D_2$ contains a different matrix of exact variable values $X_2$ and a corresponding matrix of observed noisy variables $\hat{X}_2$. I do not have any $Y$ vector in $D_2$, but I want to build a linear model that takes a noisy observation vector $\hat{x}$ and estimates $y$ (no exact $x$ will be available to predict $y$). $\endgroup$ – mbloem Mar 17 '11 at 16:26
  • $\begingroup$ "Latent variable regression" returns a lot of hits on google, but I didn't find a tutorial or some standard model. $\endgroup$ – SheldonCooper Mar 17 '11 at 20:21
  • $\begingroup$ Looking at this question may be useful: stats.stackexchange.com/questions/7772/… $\endgroup$ – SheldonCooper Mar 18 '11 at 0:09
  • $\begingroup$ @SheldonCooper Okay so I guess I would need to first use linear regression to build a model for predicting $y$ given $x$, and then a separate model for estimating the latent variable $x$ given $\hat{x}$. Any suggestions for how to build this second model with my $D_2$ data set consisting of a matrix of exact variable measurements $X_2$ and corresponding noisy variable measurements $\hat{X}_2$? $\endgroup$ – mbloem Mar 18 '11 at 16:44
1
$\begingroup$

This sounds like something very similar to a method I have seen Jerry Reiter using multiple imputation for missing data. However I can't quite remember the name of the paper. But these terms will probably be able to get you in the right(er) direction (pardon the pun).

So basically you have three variables $X$, $Y$, and $Z$. the variable $Z$ is your "gold standard" variable, and the variable $X$ is the "bronze standard" variable. You would prefer to have observed $Y$ and $Z$ together. But unfortunately, you only observe $X$ and $Y$ together, and $X$ and $Z$ together.

you can set up the model as follows. If you knew $Z$, then $X$ would be irrelevant to making inference about $Y$ (why use bronze when you've got gold?). So you have:

$$P(Y|X,Z)=P(Y|Z)$$

But you also have a model relating $X$ and $Z$, $P(Z|X)$. Now use the law of total probability, the product rule, and the above equation to expand $P(Y|X)$:

$$P(Y|X)=\int P(Y,Z|X)dZ=\int P(Y|Z,X)P(Z|X) dZ=\int P(Y|Z)P(Z|X)dZ$$

And so you have a "weighted average" of the "gold model" $P(Y|Z)$, where the weights depend on the "error model" $P(Z|X)$ (i.e. how well the bronze standard predicts the gold standard). If the $Z$ can be quite well predicted, then $P(Z|X)$ will resemble a "delta function" and the model will essentially be a "plug in" model:

$$P(Y|Z)\approx P(Y|Z)_{Z=X}$$

If $Z$ is poorly estimated from $X$, then $P(Z|X)$ will be quite "flat" and this procedure will "spread out its bets" over many different models.

Jerry Reiter's method should give some more details about the actual implementation of this.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Sounds interesting, I'll have to look into that! $\endgroup$ – Dikran Marsupial Mar 25 '11 at 16:48
1
$\begingroup$

One approach would be to use the second dataset to make a secondary model that predicts the exact variables as a function of the noisy variables and use this in operation to provide the inputs for the primary model trained on the first dataset (which predicts the target given the exact variables). However, to do this properly you would want to propogate the uncertainty of the secondary model through the primary model when making predictions. Say you used a multivariate regression model (as the uncertainty in the conditional estimates of the exact variables are unlikely to be uncorrellated) you would have a multi-variate normal distribution for the plausible values of the exact variables conditioned on the noisy ones. This could then be sampled a few thousand times and the output of the primary model averaged over that sample to get a more robust estimate for the target. I suspect if you kept everything linear and normal there would be an analytic solution so you didn't have to sample.

The regression may at least help with the bias of the noise, if nothing else.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.