In general, to use the method of least squares, a linear stochastic system is modeled as: \begin{equation} y = ax + \eta \end{equation}

where, $y$, is an observed variable, $x$ is an input while $\eta$ is error in observation. Then the parameter $a$ is estimated in the least squares sense by minimizing the error $\left(y - ax\right)^2$.

Assume that instead of modeling the system in this way, I do some change of variables and write the equation of the system as:

\begin{equation} \alpha\beta = y + \eta \end{equation}

where, $\alpha$ and $\beta$ need to be estimated. Let us also assume that I know the range of values that $\beta$ can take. Then I rewrite the estimation problem as:

\begin{equation} \alpha = \frac{y}{\beta} + \nu \end{equation}

Now suppose that I have $n$ observations i.e. $y_1, y_2, \cdots, y_n$ and I know that $\beta$ can take $m$ possible values i.e. $\beta_1, \beta_2, \cdot, \beta_m$. For each value of $\beta$, I compute the $n$ values of $\alpha$ and find variance of these $n$ values. Since $\alpha$ is constant, ideally the variance should be zero. But due to error $\nu$, the variance will be non-zero. So, the value of $\beta$ that minimizes the variance in $n$ values of $\alpha$ is the optimal estimate for $\beta$. $\alpha$ can then be found out.

Is this a correct approach to remodel the least squares problem. Potential questions can be:

  1. Why not use least squares? ( Ans: Because in some multivariate cases, $x$ can be badly conditioned.)
  2. If $y$ and $\beta$ are normally distributed then $\frac{y}{\beta}$ has Cauchy distribution and Cauchy distribution has no finite variance. So, minimizing the variance of $\frac{y}{\beta}$ does not make sense.

So, I want to know whether the modeling mentioned above makes sense.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.