Least Squares Derivation

Question

I come from physics and would like to derive the chi-square function given by the Particle Data Group:

\begin{equation} \chi^2 (\boldsymbol\theta) = (\boldsymbol y-\boldsymbol\mu(\boldsymbol \theta))^T V^{-1}(\boldsymbol y - \boldsymbol\mu(\boldsymbol \theta)), \end{equation} with $V_{ij} = cov[y_i, y_j]$ being the covariance matrice, $\boldsymbol y = (y_1, \cdots, y_N)$ being the vector of measurements and $\boldsymbol \mu(\boldsymbol \theta)$ is the vector of predicted value.

My second question would be if the nomenclature chisquare and least square have the same meaning or are these two different methods?

---

So far I started with \begin{equation} \boldsymbol{\hat y} = X \boldsymbol{\hat \beta}, \quad \text{with} \quad \boldsymbol{\hat y} = \pmatrix{y_1 \\ y_2 \\ \vdots \\y_N}, \quad X=\pmatrix{1 & x_{11} & x_{12} & \cdots & x_{1K} \\ 1 & x_{21} & x_{22} & \cdots & x_{2K} \\ \vdots & \ddots & \ddots & \ddots & \vdots \\ 1 & x_{N1} & x_{N2} & \cdots & x_{NK} }, \quad \boldsymbol{\hat \beta}=\pmatrix{\beta_0 \\ \beta_1 \\ \vdots \\ \beta_K} \end{equation} and the hat (^) marking the estimators.

From here I defined my chisquared function to minimize: \begin{equation} \chi^2 = (\boldsymbol y - \boldsymbol{\hat y})^T (\boldsymbol y- \boldsymbol{\hat y}), \end{equation} where $\boldsymbol y$ represents the given data points and $\boldsymbol{\hat y}$ the theoretically predicted estimators. The above formular looks yet similar to the one I am looking for, apart from the fact that I do not know how to introduce the inverse of the covariance matrix $V^{-1}$.

I also previously derived the relations \begin{equation} \begin{split} \boldsymbol{\hat \beta} = (X^T X)^{-1} X^T \boldsymbol{\hat y} \quad \text{and} \quad Var[\hat \beta] = \sigma(X^T X)^{-1}, \end{split} \end{equation} where I am also insecure that the approximations made to obtain the ladder identity hold true in my case.

Answer 1

Answering my question one can start from the weighted least squared, which can be summarized in "weighting" every summand of the least square sum by the inverse of variance of the the point given by \begin{equation} w_i = \frac{1}{\sigma_i^2}. \end{equation} The least square sum can then be written as: \begin{equation} \chi^2 = \sum_{i=1}^N \frac{\left(y_i-\mu(x_i; \boldsymbol{\theta})\right)^2}{\sigma^2}. \end{equation} Regarding the formula in question (in matrix form, with the covariance matrix V): \begin{equation} \chi^2 (\boldsymbol\theta) = (\boldsymbol y-\boldsymbol\mu(\boldsymbol \theta))^T V^{-1}(\boldsymbol y - \boldsymbol\mu(\boldsymbol \theta)), \end{equation} one notices that the covariance matrix V is a symmetric matrix, meaning that we can diagonalize it as \begin{equation} D = P^{-1} C P, \quad \text{with} \quad D = \pmatrix{\lambda_1 & & \\ & \lambda_2 & \\ &&\ddots & \\ &&& \lambda_N}. \end{equation} The $1/\lambda_i$ (we have to invert D) can then be interpreted as the "weights". In the case of a data with zero covariance we initially get a diagonal covariance matrix V \begin{equation} V = \pmatrix{\sigma^2_1 & & \\ & \sigma^2_2 & \\ &&\ddots & \\ &&& \sigma^2_N}, \end{equation} which recovers the method of weighted least squared. Otherways one deals with the more general form, the Generalized Least Squares method, which additionally accounts for non-zero covariances.