4
$\begingroup$

I am working on a linear regression problem using sklearn's diabetes dataset and coding things from scratch and while evaluating the performance of the model, I have come to question how to move between what the MSE is on the standardised dataset and what it would be on the unperturbed dataset.

I scaled both the X and Y variables and the algebra seems quite heavy and not sure working through it would be fruitful, hence I wanted to ask the community their thoughts.

The data are scaled according to the following formula, to give $x_{s,i}$ and $y_{s,i}$

$$\begin{align} x_{s,i} & = \frac{x_i - \bar{x}}{\sigma_{x}} \\[10pt] y_{s,i} & = \frac{y_i - \bar{y}}{\sigma_{y}} \end{align}$$

Then $\hat{\alpha}$ and $\hat{\beta}$, the MLE for the parameter's is given by $$\begin{align} \\ \hat{\alpha} & = \bar{y} - \hat{\beta} \bar{x} \\[10pt] \hat{\beta} & = \frac{s_{xy}}{s_{xx}} \\[10pt] & = \frac{\sum_{i=1}^N (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^N (x_i - \bar{x})^2} \\[10pt] \end{align}$$

With the value of the fitted value given by $$\begin{align} \hat{y} & = \hat{\alpha} + \hat{\beta}x_i \\[10pt] & = \bar{y} - \hat{\beta}\bar{x} + \hat{\beta}x_i \\[10pt] \hat{y_{i}} & = \bar{y} - \hat{\beta} (x_i - \bar{x}) \end{align}$$

If we substitute the standardised variables into this formula then it becomes

$$\begin{align} \hat{\beta_s} & = \frac{\sum_{i=1}^N (x_{s,i} - \bar{x_s})(y_{s,i} - \bar{y_s})}{\sum_{i=1}^n (x_{s,i} - \bar{x_s})^2} \\[10pt] \end{align}$$

At this point I start looking at the algebra and think I may not find a quick and clean solution to this but I still want to understand how I can go between the MSE for unscaled data, $MSE$, and then also the scaled MSE, $MSE_s$. Plugging along I get down to the following and wanted to confirm if this is worth pursuing further

$$\begin{align} MSE_s & = \frac{\sum_{i=1}^N \big(y_{s,i} - \hat{y_{s,i}}\big)^2}{N} \\[10pt] & = \frac{\sum_{i=1}^N \big(y_{s,i} - \bar{y_s} + \hat{\beta_s} (x_{s,i} - \bar{x_s})\big)^2}{N} \\[10pt] & = \frac{\sum_{i=1}^N \big(y_{s,i} - \bar{y_s} + \frac{\sum_{i=1}^n (x_{s,i} - \bar{x_s})(y_{s,i} - \bar{y_s})}{\sum_{i=1}^n (x_{s,i} - \bar{x_s})^2}(x_{s,i} - \bar{x_s})\big)^2}{N} \\[10pt] & = \frac{\sum_{i=1}^N \big(y_{s,i} - \bar{y_s} + \sum_{i=1}^n (y_{s,i} - \bar{y_s})\big)^2}{N} \\[10pt] \end{align}$$

Would be grateful for anyone's thoughts / guidance.

$\endgroup$

1 Answer 1

9
$\begingroup$

I think you're working too hard. Standardizing the predictor ($x$) variables should have no effect on the overall model fit (provided it doesn't alleviate problems with numerical instability etc.); that is, it will change the parameter estimates but not the MSE, predictions, $R^2$, AIC, etc.. Centering the response variable (or otherwise shifting it) should have no overall effect (except on the predictions, which will be shifted by an identical amount), and scaling the response variable by a factor $C$ should scale the MSE by a factor $C^2$.

If you scale $y$ by $1/s_y$ (the sample standard deviation), then the MSE should end up being scaled by $1/s_y^2$.

$\endgroup$
2
  • $\begingroup$ Thank you very much Ben. Sorry, I was not scaling by the variance - that is a typo on my part and has been corrected. Thanks for breaking that down as the shifting makes perfect sense and I was already aware of the effect of the scaling factor. One additional question I have is if I am working on a multivariate regression problem, how would I then view the scaling term as each feature has a unique standard deviation $\endgroup$ Commented Feb 25 at 18:42
  • 1
    $\begingroup$ Yes, you would scale each feature by its own standard deviation. See e.g. Schielzeth 2010 Methods in Ecology and Evolution $\endgroup$
    – Ben Bolker
    Commented Feb 25 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.