# First order condition of sum of squares with respect to variance of residuals

Consider the criterion function for ordinary least squares $$S(b)=(Y-X'b)'(Y-X'b)$$ with Y, a matrix of dependent variables, and X, a matrix of explanatory variables. It is of course known that: $$\frac{\partial S}{\partial \beta} = -2\cdot X'Y+2\cdot X'X\cdot b$$ solving for $b$ yields the OLS estimator for $b$.

Now if we think of $\sigma^2$, the variance of the residuals, as a parameter to be estimated does it then make sense to $$\frac{\partial S}{\partial \sigma^2} ?$$ if so what is it?

As usually $Y\sim \mathcal{N}(X'b,\sigma^2 I_n)$, you can write $$S(b)=S(b,\sigma)=(Y-X'b)'(Y-X'b)=\epsilon'\epsilon$$ with the vector of residuals $\epsilon \sim \mathcal{N}(0,\sigma^2 I_n)$ and their standardized version $\sigma^{-1}\epsilon=\epsilon_0 \sim \mathcal{N}(0,I_n)$.

So $$\frac{\partial S}{\partial \sigma^2}=(Y-X'b)'(Y-X'b)=\frac{\epsilon'\epsilon}{\sigma^2}={\epsilon_0}'\epsilon_0.$$

Note by the way that $$E(\frac{\epsilon'\epsilon}{\sigma^2})=E({\epsilon_0}'\epsilon_0)=n,$$ where $n$ is the length of $Y$ or $\epsilon$. This result holds even if the residuals are not Gaussian but at least centered and their variance is parametrized by $\sigma^2$.

• Thank you very much for your reply! Do you mean that the derivative is $(Y_0-X'b)'(Y_0-X'b)$ if so how would I estimate that from the data? – Henrik May 29 '14 at 11:57
• I might have been unclear, but I am pretty sure that what you call $\sigma$ is what I would like to have the derivative with respect to. – Henrik May 29 '14 at 12:22
• $\frac{\partial S}{\partial \sigma^2}$ is also a random variable. You don't estimate it, you observe it. In turn, $\frac{\epsilon'\epsilon}{\sigma^2_{\epsilon}}$ is known. It is $\frac{1}{N-r(X)}$, where $r(X)$ denotes the rank of $X$. – Horst Grünbusch May 29 '14 at 18:53
• Although what you have rewritten makes intuitive sense, Horst, from a mathematical standpoint it's going to take a lot of explanation: after all, $S$ is a random variable (not a real-valued function) whose underlying distribution is parameterized by $\sigma$. Thus you need to invoke a generalization of Calculus in which differentiating such objects makes sense and you will need to show it satisfies the usual rules. Unless you like that sort of mathematical research, you will find it easier to approach this question via the likelihood, which is a real-valued function of $b$ and $\sigma$. – whuber May 29 '14 at 21:17
• If you insist on viewing $S$ in your equations as a measurable real-valued function on an abstract probability space--which is correct but problematic for what you are trying to do--then you will run into even more trouble making sense of its derivative with respect to $\sigma$. (Recall that derivatives are limits of difference quotients: what quotient do you have in mind and how will you define the limit?) I am suggesting likelihood is a useful approach due to the intimate connection between the MLE (with Gaussian residuals) and the OLS solution. – whuber May 29 '14 at 21:50