# How to prove Mean Squarred Error (MSE)

I would like to prove this equation of Mean Squared Error (MSE):

• m is the number of training instances.
• X is a m × n matrix containing all the feature values (excluding labels) of all instances in the dataset.
• X^T is the transpose of matrix X.
• w is the model’s parameter vector, containing the bias term w0 and the feature weights w1 to wn.
• y is the vector of target values containing y(1) to y(m) values (training instances).

The above function is the gradient of the minimum squared error loss function:

where,

• m is the number of training instances.
• hw is the hypothesis function, using the model parameters w.
• x(i) is a n × 1 vector of all the feature values (excluding the label) of the i-th instance in the dataset, and y(i) is its label (the desired output for this instance)

I would really appreciate any clue for a start at least or any relevant link where I could find more information.

• I'm confident we can help, but please explain what all your symbols mean so we don't guess incorrectly.
– whuber
Commented Nov 29, 2021 at 23:07
• @whuber Yes you are right..Let me update the question Commented Nov 30, 2021 at 8:47
• Do you know matrix calculus? It's a straightforward result from that. Also, by your result, $h$ should be a linear function Commented Nov 30, 2021 at 13:50
• @Firebug I am a totally newbie in mathematics. Since you wrote matrix calculus, I will search of it. However, I would really appreciate it if you provide more input on that. Commented Nov 30, 2021 at 13:57
• @BDEngineer Firebug's done a nice job outlining the mechanics of this solution (+1). If you're looking for broader treatments, we have several threads with additional resources: stats.stackexchange.com/search?q=matrix+calculus
– Sycorax
Commented Nov 30, 2021 at 15:01

First, let's identify what we mean here:

$$\text{MSE}=\frac{1}{m}\sum (X_iW-Y_i)^2$$

And the gradient is a vector of partial derivatives:

$$\nabla_W\text{MSE}=\left[\frac{\partial \text{MSE}}{\partial W_j} \right]$$

$$\frac{\partial \text{MSE}}{\partial W_j}= \frac{1}{m}\frac{\partial}{\partial W_j}\sum (X_iW-Y_i)^2=\\ \frac{1}{m}\sum 2(X_iW-Y_i)\frac{\partial}{\partial W_j}(X_iW-Y_i)$$

Remember that $$X_iW = \sum_k x_{ik}W_{k}$$ This last derivative is then simply:

$$\frac{\partial}{\partial W_j}(X_iW-Y_i) = \frac{\partial}{\partial W_j}(\sum_k x_{ik}W_{k}-Y_i)=x_{ij}$$

Then, recognizing that the resulting sum is a vector product:

$$\frac{\partial \text{MSE}}{\partial W_j}= \frac{1}{m}\sum 2(X_iW-Y_i)x_{ij} = \frac{2}{m} x_j^T(XW-Y)$$

Where $$x_j$$ is the $$j$$-th column of $$X$$

Then reorganizing into the columns of the gradient we can recognize that it's a matrix product, arriving at the result:

$$\nabla_W\text{MSE}=\left[\frac{2}{m} x_j^T(XW-Y) \right]_j = \frac{2}{m}X^T(XW-Y)$$

Using matrix calculus:

$$\text{MSE}=\frac{1}{m}\epsilon^T\epsilon =\frac{1}{m} (XW-Y)^T(XW-Y)$$

$$\frac{\partial \text{MSE}}{\partial W}= \frac{1}{m} \frac{\partial}{\partial W}(XW-Y)^T(XW-Y)=\\ \frac{1}{m} \left(\frac{\partial (XW-Y)^T}{\partial W}(XW-Y) + \left((XW-Y)^T\frac{\partial (XW-Y)}{\partial W}\right)^T\right)=\\ \frac{1}{m} \left(X^T(WX-Y) + ((WX-Y)^TX)^T\right) = \frac{2}{m} X^T(WX-Y)$$