Placing (Bayesian) priors on variables instead of parameters?

Question

I had this question on using Bayesian analysis for situations where we believe there is measurement error (i.e. $x$ is not error-free). This goes against the traditional Frequentist and Bayesian estimation frameworks in which we believe that the distribution of $x$ is not important.

I collect some data ($x,y$) in which I believe that its possible if I had collected the data under similar circumstances, $x$ might have been different.

More formally, for observation $i = 1,...,n$, we observe pairs $(\tilde{\mathbf{x}}_i, y_i)$ where $\tilde{\mathbf{x}}_i$ represents our observed covariates. I am willing to make the assumptions that the true covariates $\mathbf{x}_i$ are normally distributed centered around the observed value and some "prior" variance I decide called $\tau$. In effect, (unlike the traditional Bayesian setting) we are placing priors on the variables themselves and not the parameters:

$$ \begin{align} y_i &= \mathbf{x}_i^\top\boldsymbol{\beta} + \epsilon_i, \quad \epsilon_i \sim N(0, \sigma^2) \\ x_{ij} &\sim N(\tilde{x}_{ij}, \tau^2_j), \quad j = 1,...,p \end{align} $$

• $\tilde{x}_{ij}$ is the observed value for covariate $j$ in observation $i$
• $x_{ij}$ is the true (unobserved) value
• $\tau^2_j$ represents our uncertainty about covariate $j$
• $\boldsymbol{\beta}$ are fixed regression parameters

Using the laws of probability, the likelihood function would be (i.e. $y$ depends on $x$, and $x$ depends on some probability distribution):

$$ \begin{align} L(\boldsymbol{\beta}, \sigma^2, \{\tau^2_j\}|\mathbf{y}, \tilde{\mathbf{X}}) &= \prod_{i=1}^n \int p(y_i|\mathbf{x}_i,\boldsymbol{\beta},\sigma^2) \prod_{j=1}^p p(x_{ij}|\tilde{x}_{ij},\tau^2_j) d\mathbf{x}_i \\ &= \prod_{i=1}^n \int \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(y_i - \mathbf{x}_i^\top\boldsymbol{\beta})^2}{2\sigma^2}\right) \\ &\quad \times \prod_{j=1}^p \frac{1}{\sqrt{2\pi\tau^2_j}}\exp\left(-\frac{(x_{ij} - \tilde{x}_{ij})^2}{2\tau^2_j}\right) d\mathbf{x}_i \end{align} $$

I see that this model might be subject to identifiability problems (i.e. impossible to tell how much much of the variance is contributed from random error vs measurement error, different combinations of parameter estimates can result in similar values of the likelihood).

To remedy this problem, I thought that the easiest thing to do would be to add a constraint:

$$ \tau^2_j \leq k_j\sigma^2 \quad \text{for all } j=1,...,p $$

Optimizing the likelihood then becomes:

$$ \begin{align} \max_{\boldsymbol{\beta}, \sigma^2, \{\tau^2_j\}} &\sum_{i=1}^n \log \int p(y_i|\mathbf{x}_i,\boldsymbol{\beta},\sigma^2) \prod_{j=1}^p p(x_{ij}|\tilde{x}_{ij},\tau^2_j) d\mathbf{x}_i \\ \text{subject to: } & \tau^2_j \leq k_j\sigma^2 \quad \text{for all } j=1,...,p \\ & \sigma^2 > 0 \\ & \tau^2_j > 0 \quad \text{for all } j=1,...,p \end{align} $$

Is this a reasonable approach to dealing with measurement error? Can we place (Bayesian) priors on variables instead of parameters in an effort to counter against measurement error problems? (Note: in this model, there will be no Posterior Distributions)

Answer 1

Setting aside the issue of identifiability for a moment, what you have proposed here seems to be the standard approach to deriving the MLE in a latent variable model representing errors-in-variables. Latent variable models are hierarchical models that typically involve a top layer showing the distribution of the observed data based on one or more "latent variables", and then an underlying layer showing the distribution of those latent variables. Both levels of the distribution typically involve some model parameters that are the subject of inference. This model form leads to a likelihood function that is an integral form to "marginalise" out the latent variable. The MLE is then typically computed using the EM algorithm.

Although your formulation here is a standard MLE for a latent variable model, it is worth noting that both classical and Bayesian models often involve the stipulation of a distribution for "latent variables". (Contrary to the assertion in your question, the Bayesian approach does not limit prior distribution only to "model parameters" unless you consider latent variables to be implicit model parameters.) Putting a prior distribution on the underlying latent variable is just be the standard treatment for a Bayesian errors-in-variables model (see e.g., Dellaportas and Stephens 1995) which treats the observed explanatory variable as being a polluted version of an underlying latent variable. If you were to use a Bayesian analysis, you would make posterior inferences using Bayes' theorem rather than deriving the MLE.

Answer 2

1. From a modelling point of view, this is a valid model that could capture interesting aspects of a particular dataset.

2. The identifiability aspect is important. I haven't taken enough time to think things through in detail, but I'm worried that injecting noise (in the model) on the predictors just results in a fundamentally similar regression model with reduced strength of the effect. If that's the case, that means that this model does not really have independent value beyond the traditional regression models.

3. In Bayesian inference, we are actually free to ignore identifiablity issues somewhat. This is an advanced point, so ignore it if it doesn't quite make sense. The model being unidentifiable has 0 impact on computing on the posterior or the joint distribution. The issues are that the model might be hard to interpret and / or might not have independent value and / or the priors might have undue influence on the result (since the data cannot resolve the unindentifiable aspects of the posterior). As long as we focus on predictions on the variables, and ignore parameters, we should be mostly fine.

4. Maximum likelihood estimation is not bayesian estimation, even though they are closely related. I'm even more worried about MLE methods in this model.

5. In Bayesian estimation, we often have issues with computing the posterior and joint distribution. These issues are compounded by the use of such models which add variables. Sometimes, it's worth it but you have to be careful.

In my opinion, in this particular case, I don't think it's worth it to expand the model in this direction, but that's only an intuition: it's not a very informed opinion.