2
$\begingroup$

EDIT: This is now written in a more minimalist form for statistics/econometrics audience rather than the more Bayesian learning or Bayesian Nash Equililbrium style.

My problem involves finding the likelihood given an observable tied down by an implicit function, which will not have any closed form solution. To set notation and the random variables:

  • Unobserved value: $Y = \bar{Y} + \epsilon_Y$, where $\epsilon_Y \sim N(0,\sigma^2_Y)$
  • An unobserved value based on $Y$, which is $y = Y + \epsilon = \bar{Y} + \epsilon_Y + \epsilon$. where $\epsilon \sim N(0,\sigma^2_y)$
  • Another unobserved value (independent of $Y$), $x = \bar{x} + \nu$, where $\nu \sim N(0,\sigma^2_x)$
  • All of the noise terms, $\epsilon, \epsilon_Y,$ and $\nu$ are independent
  • Finally, assume there exists a $F(y,x,p)$ with reasonable convexity, monotonicity, invertibility, etc. but which does not have an explicit formula. Furthermore, the following must hold for any $y, x$ and $p$. $$ F(y, x, p) = 0 $$
  • $p$ is the only observable

Let's say we can form a joint distribution: $Z \equiv \begin{bmatrix} y & x & Y & p \end{bmatrix}$. Then partition this as $Z \equiv \begin{bmatrix} Z_1 & Z_2 \end{bmatrix}$ where $Z_1 \equiv \begin{bmatrix}y & x & Y\end{bmatrix}$ and $Z_2 \equiv \begin{bmatrix}p\end{bmatrix}$.

Question: Given knowledge of $F(\cdot)$, and an observation of $p$, what is the likelihood of $y, x$ and $Y$? i.e., $$ L(Y, y, x |p) $$

From the connection between densities and likelihood, isn't the likelihood the same thing (or at least proportional to) the conditional distribution of: $ Z_1 \,|\, Z_2 $, which can be calculated from the joint distribution of $Z$?


A few things:

  • Note that there are two unobserved values here, $y$ and $x$. This means that the likelihood for $y$ is non-degenerate.
    • i.e. if there was no $x$, then the $F(y,p) = 0$ could be inverted to find $y$ exactly. Then the likelihood of $Y$ is easy.
    • Hence, I don't think this problem can be reduced to anything much simpler
  • I have successfully done this in closed form with a linear $F(\cdot)$ using the projection theorem, but in general I have a non-linear $F(\cdot)$ (which is actually determined by this likelihood after I combine with a prior on $Y,y,x$. This is a tricky Bayesian Learning Problem with a fixed-point).
  • Hence, numerical methods will almost certainly be necessary, but I can't design them until I have at least some implicit function for the likelihood based on derivatives, inverses, etc. of the likelihood.

For the Linear Case

To help understand the setup, let me sketch out the approach for the case where $F(\cdot)$ is affine.

(1) Assume that $F(x,y,p) = -p + c_1 + c_2 x + c_3 y$ for some $c_1,c_2, c_3$. Then rearranging and inverting the $F(x,y,p) = 0$ condition you get $$p = c_1 + c_2 x + c_3 y $$ (2) This can also be written as a constant term and then some constants on the shocks. $$ p = \left[c_1 + c_2 \bar{x} + c_3 \bar{Y}\right] + c_2 \nu + c_3 \epsilon_Y + c_3 \epsilon $$

(3) From here, note that $p$ is written in terms of constants and the shocks, given $\epsilon, \epsilon_Y,$ and $\nu$. We also have $Y = \bar{Y} + \epsilon_Y$, $y = \bar{Y} + \epsilon_Y + \epsilon$, and $x = \bar{x} + \nu$

So using the distributions of the shocks, we can form a joint distribution of $\begin{bmatrix} y & x & Y & p \end{bmatrix}$ using the variance and covariances of all of these normal variables, and due to linearity it remains gaussian. I won't write out all those steps as they are standard but tedious

Lets say that we can write out the joint distribution partitioned as, $$ \begin{bmatrix} Z_1 \\ Z_2 \end{bmatrix} \sim N\left(\begin{bmatrix}\mu_1 \\ \mu_2\end{bmatrix}, \begin{bmatrix}\Sigma_{11} & \Sigma_{12}\\ \Sigma_{21} & \Sigma_{22}\end{bmatrix} \right) $$ where $Z_1 \equiv \begin{bmatrix}y & x & Y\end{bmatrix}$ and $Z_2 \equiv \begin{bmatrix}p\end{bmatrix}$.

(4) Finally, you can use the conditional distribution of the multivariate normal from https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Conditional_distributions The general formula is, $$ Z_1 \,|\, Z_2 \sim N\left( \mu_1 + \Sigma_{12} {\Sigma_{22}}^{-1} \left( Z_2 - \mu_2 \right) , \Sigma_{11} - \Sigma_{12}{\Sigma_{22}}^{-1}\Sigma_{21} \right) $$

(5) We are done as $Z_2 = p$ is the observable and we can calculate the means and variances of the joint distribution. One difficulty: If the joint distribution is non-Gaussian (and it would be for a general $F$), then there may be some sort of projection theorem required for more generality to solve this problem.

$\endgroup$
  • $\begingroup$ You're using $\bar Y$ do denote an unobservable quantity (in this case a parameter); if you must do so (rather than follow more usual convention of using bars for sample averages), it's best to highlight that you've done so. You should probably define what "hidden private values" are. $\endgroup$ – Glen_b Sep 15 '15 at 22:11
  • $\begingroup$ Sorry, this is the standard convention for bayesian learning with multiple agents. Let me reorganize this to be more consistent with how the statistics crew thinks about this. $\endgroup$ – jlperla Sep 16 '15 at 16:28
  • $\begingroup$ @Glen_b The text has been updated and simplified to make this easier for a non-learning audience. $\endgroup$ – jlperla Sep 16 '15 at 16:55
  • $\begingroup$ I don't mind notation outside the usual stats convention (as you say, some areas of work do things different ways, especially in ML but also in other areas), as long as it's not likely to mislead almost everyone else (i.e. as long as the main convention-breaking aspects of notation are clearly defined, even if they're fairly conventional in the scope of the question). $\endgroup$ – Glen_b Sep 16 '15 at 19:01
  • 1
    $\begingroup$ Thanks. Changed to independent. There is nothing at all ``sketchy'' going on with the noise terms, so hopefully that is sufficient. $\endgroup$ – jlperla Sep 16 '15 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.