1
$\begingroup$

I need to calculate the probability of $P(L,D)$, where $L$ and $D$ are not independent. I have estimated $P(L)$ and $P(D)$ with two distinct models and I also know $P(L|D)$ and $P(D|L)$.

As far as I can see it I have two different estimates of $P(L,D)$:

$P(L,D) =P(L) * P(D|L)$

$P(L,D) =P(D) * P(L|D)$

Am I best off just averaging these two estimates for the joint $P(L,D)$?

Improve this question
$\endgroup$
8
  • $\begingroup$ The question as posed is nice and straightforward to read, but some other info it would be useful to add in at the end: 1) What kind of variables are L and D? Continuous? Categorical? What values can they take? 2) Given know P(L|D) and P(D|L) exactly, could you tell us what they are? 3) What models have you used to estimate P(L) and P(D)? Why? 4) What data do you have access to? It sounds like you have samples from P(L) and samples from P(D), to which you have fitted a model. If so, how many samples from each? 5) What are you going to do with P(L,D) once you know it? $\endgroup$ – Pat Oct 16 '13 at 11:41
  • $\begingroup$ And why I would like to know: 1) To get an idea of how much missing information there is to infer, given you knowledge of the two conditional distributions. 2) Similar reasons to 1. 3) If you have good reason to believe one model's estimates are closer to the 'true' distribution, you should favour using it . 4) If you have more samples of L than D, for example, estimates of P(L) may be more accurate. 5) Depending on the intended application, it may be best to keep both estimates of P(L,D). $\endgroup$ – Pat Oct 16 '13 at 11:49
  • $\begingroup$ 1) What kind of variables are L and D? Continuous? Categorical? What values can they take? They are continuous probabilities, so can take values between 0 amd 1? 2) Given know P(L|D) and P(D|L) exactly, could you tell us what they are? P(L|D) = 0.385, P(D|L) = 0.735 3) What models have you used to estimate P(L) and P(D)? Why? $\endgroup$ – Rob Sedgwick Oct 16 '13 at 12:48
  • $\begingroup$ I have two linear regression models to estimate each value. I want to estimate whether things and L and D are the same time. I could build a third model but wondered if I could use the two I have. 4) What data do you have access to? It sounds like you have samples from P(L) and samples from P(D), to which you have fitted a model. If so, how many samples from each? I have the same set of samples for each model. That's the historical data to calculate the regression values. $\endgroup$ – Rob Sedgwick Oct 16 '13 at 12:48
  • $\begingroup$ 5) What are you going to do with P(L,D) once you know it? – Pat 52 mins ago I am going to select those with the highest probablity from a current set of data where neither L or D are known. I want to select the things which are most likely to be L and D at the same time. I am inclined to keep both estimates and average them. $\endgroup$ – Rob Sedgwick Oct 16 '13 at 12:49
4
$\begingroup$

If you have two estimators, call them $\hat{\beta}_1$ and $\hat{\beta}_2$, of the same parameter $\beta$, then you can combine them in a variety of ways. Let's suppose you know that the two estimators are consistent and asymptotically normal --- this is generally true of estimators you get from maximum likelihood methods, method of moments methods, and some other methods as well. Furthermore, suppose you know the (asymptotic) variances of the two estimators, $V(\hat{\beta}_1)$ and $V(\hat{\beta}_2)$ and the covariance of the two estimators $Cov(\hat{\beta}_1,\hat{\beta}_2)$.

You propose a combined estimator of $\beta$ given by $\frac{1}{2}\hat{\beta}_1+\frac{1}{2}\hat{\beta}_2$. This is consistent and asymptotically normal if $\hat{\beta}_1$ and $\hat{\beta}_2$ are. What is its variance? \begin{align} V(\frac{1}{2}\hat{\beta}_1+\frac{1}{2}\hat{\beta}_2) &= \frac{1}{4}V(\hat{\beta}_1) +\frac{1}{4}V(\hat{\beta}_2) +\frac{1}{2}Cov(\hat{\beta}_1,\hat{\beta}_2) \end{align}

This estimator might be better (lower variance) or worse than either $\hat{\beta}_1$ or $\hat{\beta}_2$. If, say, $\hat{\beta}_1$ has a crazy-high variance, then the variance of $\frac{1}{2}\hat{\beta}_1+\frac{1}{2}\hat{\beta}_2$ might be higher than the variance of $\hat{\beta}_2$. We would like to avoid this. Also, we would like to find the best (i.e. lowest variance) way of combining the two estimators while preserving consistency. That is, we want to solve: \begin{align} &min_{\lambda}V(\lambda\hat{\beta}_1+(1-\lambda)\hat{\beta}_2)\\ &min_{\lambda}\lambda^2V(\hat{\beta}_1)+(1-\lambda)^2V(\hat{\beta}_2) +2\lambda(1-\lambda)Cov(\hat{\beta}_1,\hat{\beta}_2) \end{align} The first order condition is: \begin{align} 2\lambda V(\hat{\beta}_1)-2(1-\lambda)V(\hat{\beta}_2) +2(1-2\lambda)Cov(\hat{\beta}_1,\hat{\beta}_2)&=0 \\ \frac{V(\hat{\beta}_2)-Cov(\hat{\beta}_1,\hat{\beta}_2)}{V(\hat{\beta}_1)+V(\hat{\beta}_2)-2Cov(\hat{\beta}_1,\hat{\beta}_2)} &= \lambda \end{align} The combined estimator is then: \begin{align} \hat{\beta}_* = \frac{V(\hat{\beta}_2)-Cov(\hat{\beta}_1,\hat{\beta}_2)}{V(\hat{\beta}_1)+V(\hat{\beta}_2)-2Cov(\hat{\beta}_1,\hat{\beta}_2)} \hat{\beta}_1 + \frac{V(\hat{\beta}_1)-Cov(\hat{\beta}_1,\hat{\beta}_2)}{V(\hat{\beta}_1)+V(\hat{\beta}_2)-2Cov(\hat{\beta}_1,\hat{\beta}_2)} \hat{\beta}_2 \end{align} Because of the way we set up $\lambda$ in the minimization, you are assured that this new estimator is consistent. Notice, if the covariance between the two estimators is zero, then this new estimator is just the weighted sum of the two original estimators where the original estimators are weighted inversely to their variance---the new estimator "pays attention" to the old estimators inversely according to their variances.

Finally, the variance of the new estimator is: \begin{align} V(\hat{\beta}_*) = \frac{V(\hat{\beta}_1)V(\hat{\beta}_2)-2Cov^2(\hat{\beta}_1,\hat{\beta}_2)}{V(\hat{\beta}_1)+V(\hat{\beta}_2)-2Cov(\hat{\beta}_1,\hat{\beta}_2)} \end{align}

This process of optimally combining multiple estimators is called "minimum distance estimation" in econometrics. A cite is chapter 13 in Greene, Econometric Analysis, seventh ed.

Improve this answer
$\endgroup$
2
$\begingroup$

The answer depends on the answer to this question:

"Why are (P(L) * P(D|L)) and (P(D) * P(L|D)) different?".

If it depends on the fact that the model for P(L|D) is unreliable you should only keep (P(L) * P(D|L)), if you have no idea you can consider taking the average.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I generate P(L) and P(D) from two different models. I do not know which is best. $\endgroup$ – Rob Sedgwick Oct 16 '13 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.