How does the covariance matrix of a fit get computed? I often have to fit data of physical experiments as a student. I always use (python's) functions like numpy.polyfit or scipy.optimize.curve_fit for that purpose. They also allow you to retrieve the covariance matrix of the parameters which has the variances of the parameters on its diagonal.
This could be a rather basic question but I do not understand why there are non-zero variances at all. I thought the fit is well-defined because it is the fit with the least squares so there should be no variance. What do these variances of the parameters tell me? Do they come from numerical errors?
 A: I think you're confusing two different concepts in a linear regression exercise:


*

*"features scaling" : the process of taking each of your features and modifying them to have a null Expected value and a null Variance

*"fitting": iterating over a fitting algorithm (like gradient descent) to find the best vector (often called $\theta$) which will give you the smallest for the smallest "mean square error" (the sum of the squared difference between your estimation and the real value). This is what numpy.polyfit does ("poly" because it add polynomial features).
Features scaling aims at accelerating the convergence of the "fitting algorithm" 
Fitting aims at... Fitting (!!!) the vector which ought to minimize the mean square error.
So: 


*

*"feature scaling" transforms the features $X_i$ into new $X'_i$ whose variances are equal to zero.

*"fitting" finds the vector that minimize the mean square error through multiple iteration but without touching the variance.


Moreover, the covariance matrix of $n$ features measures the "link" between the $n$ different features and display this link visually in a $n\times n$ matrix.
Let say you have a set of $m$ individuals $X_1, X_2, ..., X_m$
Each of these individual has 3 observed characteristics (or features): age, size, gender.
The covariance matrix $\Sigma$ will be a $3\times 3$ matrix (as there are $3$ features) describing the covariance between these $3$ features on this set of $m$ individuals:
$$
\Sigma=\begin{bmatrix}
\sigma_{age,age}^2 & \sigma_{age, size}^2 & \sigma_{age,gender}^2 \\
\sigma_{age, size}^2 & \sigma_{size, size}^2 & \sigma_{size,gender}^2 \\
\sigma_{age, gender}^2 & \sigma_{size, gender}^2 & \sigma_{gender, gender}^2
\end{bmatrix}
$$
If two features are independent from each other (like the age and the gender might be), their covariance will be equal to zero and you will find a zero on both side of the diagonal (as $\Sigma$ is obviously symmetric: cov(X,Y) = cov(Y,X)). 
If two features are not independent (like age and size might be), the covariance won't be equal to zero.
Last: 


*

*if two features and dependent and "moves together" on the same direction (i.e. if $X$ increase, $Y$ increases too), then $Cov(X,Y)$ will be strictly positive.

*if two features and dependent and "moves together" on the opposite direction (i.e. if $X$ increase, $Y$ decreases), then Cov(X,Y) will be strictly negative.
