# Toy regression question with latent variables

I originally asked this on a machine learning site, but one of the responses made me think that maybe this site is more suitable.

Suppose you have two weighted coins, and every day you flip each one a number of times and record the total number of heads. So on the tenth day you might have flipped coin A 106 times, coin B 381 times, and recorded 137 heads. Supposing your goal is to figure out the weights of each coin, is it reasonable to just regress the number of heads on the number of flips for each coin? E.g, something along the lines of:

num_heads ~ num_flips_A + num_flips_B + intercept

However,it doesn't seem to make sense to have an intercept term in this scenario (it is negative for my data, which is confusing), so I tried subtracting out -1 from the formula, and it seemed to yield reasonable results. My first question is whether this approach is a good one.

Now, suppose that you suspect the existence of a third coin, C, that someone else is flipping unbeknownst to you, and the heads for that coin are getting mixed up in your count. The number of flips for this coin are not recorded, but you do not particularly care about its weight - it's more of a confounding factor. Then would it be reasonable to fit a similar regression, but constrain the intercept to be positive?

Thanks for any help

-

Let's take the second question first: in most cases the presence of C makes the parameters unidentifiable and there's no way to estimate anything. Constraining the intercept to be positive won't help at all.

The first problem can be solved by Maximum Likelihood. After all, writing $p$ for the probability that a flip of A is heads, $q$ for the probability that B is heads, and $k_i$, $n_i$, and $m_i$ for the numbers of heads, number of flips of A, and number of flips of B in the the $i^\text{th}$ toss, respectively, we know

$$\Pr[k_i] = \sum_{j=0}^{k_i}{{{n_i}\choose{j}}{{m_i}\choose{k_i-j}}p^j(1-p)^{n-j}q^{k_i-j}(1-q)^{m_i-k_i-j}}$$

(taking ${r}\choose{s}$ to be zero whenever $r \lt s$ or $s \lt 0$). The product of these expressions (over $i$) is the likelihood to be maximized by varying $p$ and $q$ subject to $0 \le p, q \le 1$.

Maximizing this likelihood for large amounts of data does not look appetizing. If we assume both $p$ and $q$ are sufficiently far from $0$ and $1$ and that the numbers of coin flips are usually large enough that all of $n_i p$, $n_i(1-p)$, $m_i q$, and $m_i (1-q)$ exceed $5$ or so, we can apply the Normal approximation to the Binomial. This tells us that, to a good approximation, the distribution of $k_i$ is Normal with mean $p n_i + q m_i$ and variance $p(1-p) n_i + q(1-q) m_i$. Thus, weighted least squares regression (WLS) of $k$ on $(n,m)$ without any intercept term is justified. The weights are inversely proportional to these variances. However, the variances depend on the solution. Several approaches immediately come to mind:

1. Use ML based on the Normal approximation. This likelihood is not as messy as the (more accurate) Binomial likelihood.

2. Guess initial values for $(p, q)$, use these to compute the weights, and perform WLS to update the estimates of $(p, q)$. Replace the initial values with these estimates and iterate until convergence occurs (one hopes).

3. Obtain Bayesian estimates for the parameters.

## Edit

Assuming, as suggested in a comment, that C is flipped approximately the same number of times (say $l$) in each trial, let's analyze this situation using the Normal approximation. Letting the expectation of C be $r$, the expectation of $k_i$ becomes $p n_i + q m_i + r l$ with variance $p(1-p)n_i + q(1-q)m_i + r(1-r)l$. We can consider playing the iterated WLS game by estimating $p$, $q$, and $\rho = r l$ with WLS (constraining the intercept to be nonnegative as originally suggested) and updating the previous values of the parameters with the new estimates $\hat{p}$, $\hat{q}$, and $\hat{\rho} / l$. Thus, when $l$ is known, it appears that $p$, $q$, and $r$ are indeed identifiable and that--assuming the iterative method works--there is an effective algorithm for finding them. Note, too, that the Normal approximation should work well provided the expectation of the $k_i$ is typically between 5 and $n_i + m_i + l - 5$; that is, some of $p$, $q$, and $r$ can be quite small or even zero and this still should work provided that neither heads nor tails is a rare occurrence among the totals.

When $l$ is not known or varies we can still estimate $r l$ provided we have a reasonable estimate for the contribution of C to the variance, $r(1-r)l$. "Reasonable" means that when we vary $l$ and $r$ within meaningful ranges, the changes in the total variances $p(1-p)n_i + q(1-q)m_i + r(1-r)l$ do not alter the parameter estimates appreciably. This would be the case, for instance, when the contribution of C to the total number of heads is consistently relatively small. (We don't really care about $r l$--it's a nuisance parameter--but if it is not accurately estimated then there must be some bias in the estimation of $p$ and $q$.)

-
Thanks for your reply. I've done (1) and it seems pretty reasonable. As for dealing with the presence of C - I neglected to mention that the number of flips, or number of heads, of C should be fairly constant, which is why I thought including the intercept term might work. –  grg s Nov 17 '10 at 23:17