1
$\begingroup$

I would like to prove this equation of Mean Squared Error (MSE):

enter image description here

  • m is the number of training instances.
  • X is a m × n matrix containing all the feature values (excluding labels) of all instances in the dataset.
  • X^T is the transpose of matrix X.
  • w is the model’s parameter vector, containing the bias term w0 and the feature weights w1 to wn.
  • y is the vector of target values containing y(1) to y(m) values (training instances).

The above function is the gradient of the minimum squared error loss function:

enter image description here

where,

  • m is the number of training instances.
  • hw is the hypothesis function, using the model parameters w.
  • x(i) is a n × 1 vector of all the feature values (excluding the label) of the i-th instance in the dataset, and y(i) is its label (the desired output for this instance)

I would really appreciate any clue for a start at least or any relevant link where I could find more information.

$\endgroup$
7
  • 4
    $\begingroup$ I'm confident we can help, but please explain what all your symbols mean so we don't guess incorrectly. $\endgroup$
    – whuber
    Nov 29, 2021 at 23:07
  • $\begingroup$ @whuber Yes you are right..Let me update the question $\endgroup$
    – BDEngineer
    Nov 30, 2021 at 8:47
  • $\begingroup$ Do you know matrix calculus? It's a straightforward result from that. Also, by your result, $h$ should be a linear function $\endgroup$
    – Firebug
    Nov 30, 2021 at 13:50
  • $\begingroup$ @Firebug I am a totally newbie in mathematics. Since you wrote matrix calculus, I will search of it. However, I would really appreciate it if you provide more input on that. $\endgroup$
    – BDEngineer
    Nov 30, 2021 at 13:57
  • 1
    $\begingroup$ @BDEngineer Firebug's done a nice job outlining the mechanics of this solution (+1). If you're looking for broader treatments, we have several threads with additional resources: stats.stackexchange.com/search?q=matrix+calculus $\endgroup$
    – Sycorax
    Nov 30, 2021 at 15:01

1 Answer 1

2
$\begingroup$

First, let's identify what we mean here:

$$\text{MSE}=\frac{1}{m}\sum (X_iW-Y_i)^2$$

And the gradient is a vector of partial derivatives:

$$\nabla_W\text{MSE}=\left[\frac{\partial \text{MSE}}{\partial W_j} \right]$$

$$\frac{\partial \text{MSE}}{\partial W_j}= \frac{1}{m}\frac{\partial}{\partial W_j}\sum (X_iW-Y_i)^2=\\ \frac{1}{m}\sum 2(X_iW-Y_i)\frac{\partial}{\partial W_j}(X_iW-Y_i) $$

Remember that $X_iW = \sum_k x_{ik}W_{k}$ This last derivative is then simply:

$$\frac{\partial}{\partial W_j}(X_iW-Y_i) = \frac{\partial}{\partial W_j}(\sum_k x_{ik}W_{k}-Y_i)=x_{ij}$$

Then, recognizing that the resulting sum is a vector product:

$$\frac{\partial \text{MSE}}{\partial W_j}= \frac{1}{m}\sum 2(X_iW-Y_i)x_{ij} = \frac{2}{m} x_j^T(XW-Y) $$

Where $x_j$ is the $j$-th column of $X$

Then reorganizing into the columns of the gradient we can recognize that it's a matrix product, arriving at the result:

$$\nabla_W\text{MSE}=\left[\frac{2}{m} x_j^T(XW-Y) \right]_j = \frac{2}{m}X^T(XW-Y)$$


Using matrix calculus:

$$\text{MSE}=\frac{1}{m}\epsilon^T\epsilon =\frac{1}{m} (XW-Y)^T(XW-Y)$$

$$\frac{\partial \text{MSE}}{\partial W}= \frac{1}{m} \frac{\partial}{\partial W}(XW-Y)^T(XW-Y)=\\ \frac{1}{m} \left(\frac{\partial (XW-Y)^T}{\partial W}(XW-Y) + \left((XW-Y)^T\frac{\partial (XW-Y)}{\partial W}\right)^T\right)=\\ \frac{1}{m} \left(X^T(WX-Y) + ((WX-Y)^TX)^T\right) = \frac{2}{m} X^T(WX-Y)$$

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.