How does the covariance matrix of a fit get computed?

I often have to fit data of physical experiments as a student. I always use (python's) functions like numpy.polyfit or scipy.optimize.curve_fit for that purpose. They also allow you to retrieve the covariance matrix of the parameters which has the variances of the parameters on its diagonal.

I'm not really good at statistics and this could be a very stupid question but I do not understand why there are non-zero variances at all. I thought the fit is well-defined because it is the fit with the least squares so there should be no variance. What do these variances of the parameters tell me? Do they come from numerical errors?

I think you're confusing two different concepts in a linear regression exercise:

• "features scaling" : the process of taking each of your features and modifying them to have a null Expected value and a null Variance

• "fitting": iterating over a fitting algorithm (like gradient descent) to find the best vector (often called $\theta$) which will give you the smallest for the smallest "mean square error" (the sum of the squared difference between your estimation and the real value). This is what numpy.polyfit does ("poly" because it add polynomial features).

Features scaling aims at accelerating the convergence of the "fitting algorithm"

Fitting aims at... Fitting (!!!) the vector which ought to minimize the mean square error.

So:

• "feature scaling" transforms the features $X_i$ into new $X'_i$ whose variances are equal to zero.
• "fitting" finds the vector that minimize the mean square error through multiple iteration but without touching the variance.

Moreover, the covariance matrix of $n$ features measures the "link" between the $n$ different features and display this link visually in a $n\times n$ matrix.

Let say you have a set of $m$ individuals $X_1, X_2, ..., X_m$

Each of these individual has 3 observed characteristics (or features): age, size, gender.

The covariance matrix $\Sigma$ will be a $3\times 3$ matrix (as there are $3$ features) describing the covariance between these $3$ features on this set of $m$ individuals:

$$\Sigma=\begin{bmatrix} \sigma_{age,age}^2 & \sigma_{age, size}^2 & \sigma_{age,gender}^2 \\ \sigma_{age, size}^2 & \sigma_{size, size}^2 & \sigma_{size,gender}^2 \\ \sigma_{age, gender}^2 & \sigma_{size, gender}^2 & \sigma_{gender, gender}^2 \end{bmatrix}$$

If two features are independent from each other (like the age and the gender might be), their covariance will be equal to zero and you will find a zero on both side of the diagonal (as $\Sigma$ is obviously symmetric: cov(X,Y) = cov(Y,X)).

If two features are not independent (like age and size might be), the covariance won't be equal to zero.

Last:

• if two features and dependent and "moves together" on the same direction (i.e. if $X$ increase, $Y$ increases too), then $Cov(X,Y)$ will be strictly positive.

• if two features and dependent and "moves together" on the opposite direction (i.e. if $X$ increase, $Y$ decreases), then Cov(X,Y) will be strictly negative.

• Hi, I've added Mathjax (basically, LATEX) formatting to your question to render math. Please check that it says what you mean it to. You can find a mathjax tutorial here. meta.math.stackexchange.com/questions/5020/… – Sycorax May 6 '15 at 14:26
• Hi tmangin and user777, thank you a lot for your detailed explanation and for taking time! Actually a friend of mine just solved my problem. It's hard to explain my actual problem and i think it wouldn't help anybody. It's basically because i didn't understand some parts of the concept behind fitting. So i think we should just left it at that. Thanks again! – user76402 May 6 '15 at 21:39