Suppose you fit a linear/logistic regression $g(y) = a_0 + a_1\cdot x_1 + a_2\cdot x_2$, with the aim of an unbiased estimate of $\frac{a_1}{a_2}$. You are very confident that both $a_1$ and $a_2$ are very positive relative to the noise in their estimations.

If you have the joint covariance of $a_1, a_2$, you could calculate, or at least simulate the answer. Are there any better ways, and in real-life problems with a lot of data, how much trouble do you get in for taking the ratio of estimates, or for taking a half-step and assuming the coefficients are independent?

  • 1
    $\begingroup$ In logistic regression as described, how do you find an unbiased estimator of $a_0$ or $a_1$? The problem is unrelated with the correlation between the coefficients. $\endgroup$
    – Xi'an
    Sep 18 '16 at 13:54
  • 6
    $\begingroup$ Something to ponder: What if one or both of the coefficients were zero? $\endgroup$
    – cardinal
    Sep 18 '16 at 19:59
  • $\begingroup$ Yeah, good point. I'm implicitly assuming that both coefficients are sufficiently positive that there's no danger of noise leading to crossed signs (re: andrewgelman.com/2011/06/21/inference_for_a). I'll edit. $\endgroup$
    – quasi
    Sep 18 '16 at 22:33
  • 2
    $\begingroup$ How precisely do you estimate $a_1$ and $a_2$ in your regression? Is a consistent estimator with small standard errors sufficient? Is it important that your estimator is unbiased? Would it work for your application to just take $\frac{\hat{a}_1}{\hat{a}_2}$ and calculate the standard-error for that using the delta method and the estimated covariance matrix for $(a_1, a_2)$ from your regression. $\endgroup$ Sep 18 '16 at 23:07
  • 1
    $\begingroup$ Have you considered Fieller's theorem? Look here: stats.stackexchange.com/questions/16349/… $\endgroup$
    – soakley
    Sep 19 '16 at 21:36

I would suggest doing error propagation on the variable type and minimize either the error or relative error of $\frac{a_1}{a_2}$. For example, from Strategies for Variance Estimation or Wikipedia

$f = \frac{A}{B}\,$
$\sigma_f^2 \approx f^2 \left[\left(\frac{\sigma_A}{A}\right)^2 + \left(\frac{\sigma_B}{B}\right)^2 - 2\frac{\sigma_{AB}}{AB} \right]$

$\sigma_f \approx \left| f \right| \sqrt{ \left(\frac{\sigma_A}{A}\right)^2 + \left(\frac{\sigma_B}{B}\right)^2 - 2\frac{\sigma_{AB}}{AB} }$

As a guess, you probably want to minimize $(\frac{\sigma_f}{f})^2$. It is important to understand that when one does regression to find a best parameter target, one has forsaken goodness of fit. The fit process will find a best $\frac{A}{B}$, and this is definitively not related to minimizing residuals. This has been done before by taking logarithms of a non-linear fit equation, for which multiple linear applied with a different parameter target and Tikhonov regularization.

The moral of this story is that unless one asks the data to yield the answer that one desires, one will not obtain that answer. And, regression that does not specify the desired answer as a minimization target will not answer the question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.