Estimating the errors in parameters in the ordinary least square

Question

I am reading the book An Introduction to Error Analysis by John R. Taylor. In Ch8: Least-Squares Fitting, he has derived expressions for parameters $A$ and $B$ in fitting the line $A+Bx$ to the set of $N$ observations $(x_i, y_i)$. The maximum likelihood estimates of the parameters are:

$$A =\frac{\sum x^2\sum y-\sum x\sum xy}{\Delta}$$ $$B = \frac{N\sum xy-\sum x\sum y}{\Delta}$$ where $$\Delta = N\sum x^2 - \left(\sum x\right)^2$$

I have verified these to be correct. However, now he argues that the errors in the estimation for $A$ and $B$ can be found by simple error propagation method since errors in all observation $y_i$ are assumed to be $\sigma_y$ (i.e. all the errors are equal), while $x_i$ have no errors in them. He then claims that this should give the following errors on $A$ and $B$:

$$\sigma_A = \sigma_y\sqrt{\frac{\sum x^2}{\Delta}}$$ and $$\sigma_B = \sigma_y\sqrt{\frac{N}{\Delta}}$$

However, my calculations don't match with these. The way I calculate error in $A$ for example is this: since there are no errors on $x$, all the terms that contain only $x$ remain unaffected (e.g. $\sum x$ and $\Delta$). The uncertainty in $xy$ is $x\sigma_y$ and because all $y$ are distributed normally, I can get the uncertainty in $\sum xy$ by adding these uncertainties in quadrature: $\sqrt{\sum x^2\sigma_y^2}=\sigma_y\sqrt{\sum x^2}$. Similarly the uncertainty in $\sum x^2\sum y$ is $\sigma_y \sqrt{N}\sum x^2$. Now here I face the first problem: How do we combine the errors in the two terms in the numerator of $A$? Since they are not independent, I feel that I cannot add their errors in quadrature. But adding the errors directly leads me nowhere close to what is given in the book. So let's just add the errors in quadrature to get the uncertainty in $A$ as:

$$\sigma_A = \frac{1}{\Delta}\sqrt{\left(\sigma_y\sqrt{N}\sum x^2\right)^2+\left(\sum x\sigma_y\sqrt{\sum x^2}\right)^2}$$ $$\sigma_A = \frac{1}{\Delta}\sigma_y\sqrt{\sum x^2}\sqrt{N\sum x^2 + \left(\sum x\right)^2 }$$

Notice the similarity in the square root expression above and the expression for $\Delta$ given at the start. If it were not for the plus sign inside the square root, the expression would have been exactly equal to $\Delta$ leading to immediate cancellation of $\sqrt{\Delta}$ and we would have been left with the expression for $\sigma_A$ given in the book! However the plus sign messes up everything. (And this is my second problem!). What am I exactly missing?

Answer 1

in fact i don't exactly understand what you say so i will just give a simple answer. First i write $Var(X)=\mathbb{E}((X-\mathbb{E}(X))^2)$ for the variance of any random variable $X$.

I guess you know it but $Var(X-Y) = Var(X)+Var(Y)-2*Cov(X,Y)$ where $Cov$ stands for covariance.(for me you just forget the minus covariance in your formula but i write all to be exhaustive).

Also $Var(\alpha X) = \alpha^2 Var(X)$. From there we have

$A = \alpha (X-Y)$ with $\alpha=\frac{1}{\Delta}$ and $X=\gamma \sum y$ and $Y=\beta \sum xy$ where $\gamma = \sum x^2$ and $\beta=\sum x$.

Also we have $Var(\sum y) = N \sigma_y^2$ (i suppose according to your calculation and notation we have N samples independent).

And we have as you wrote $Var(\sum xy) = \sum x^2 \sigma_y^2$.

We also have (and this is where writing index can really be handy) $Cov(\sum_i x_iy_i, \sum_j y_j) = \sum_i Cov(\sum_j x_jy_j, y_i) = \sum_i \sum_j x_j Cov(y_j,y_i) = \sum_i x_i Var(y_i)$ since the term where $i=j$ stays. With your notation $Cov(X,Y) = \sigma_y^2 \sum x$

So we get $$ \sigma_A^2 = \alpha^2 (Var(X)+Var(Y)-2Cov(X,Y)) $$

$$ \sigma_A^2 = \frac{1}{\Delta^2} (N \sigma_y^2 \gamma^2+\beta^2\sum x^2 \sigma_y^2-2\sigma_y^2 \gamma \beta \sum x) $$

Facotrizing by $\sigma_y^2$ and replacing $\gamma$ and $\beta$ $$ \sigma_A^2 = \frac{\sigma_y^2}{\Delta^2} \left(N(\sum x^2)^2+(\sum x^2)(\sum x)^2-2(\sum x)^2(\sum x^2)\right) $$ $$ \sigma_A^2 = \frac{\sigma_y^2}{\Delta^2} \left(N(\sum x^2)^2-(\sum x^2)(\sum x)^2\right) $$ Factorizing by $\sum x^2$ $$ \sigma_A^2 = \frac{\sigma_y^2 \sum x^2}{\Delta^2} \left(N(\sum x^2)-(\sum x)^2\right) $$ and we have $\Delta = N(\sum x^2)-(\sum x)^2$ so we get the result $$ \sigma_A^2 = \frac{\sigma_y^2 \sum x^2}{\Delta} $$