6
$\begingroup$

I am comparing the mean squared error (MSE) from a standard OLS regression with the MSE from a ridge regression. I find the OLS-MSE to be smaller than the ridge-MSE. I doubt that this is correct. Can anyone help me finding the mistake?

In order to understand the mechanics, I am not using any of Matlab's build-in functions.

% Generate Data. Note the high correlation of the columns of X. 
    X = [3, 3
        1.1 1
        -2.1 -2
        -2 -2]; 
    y =  [1 1 -1 -1]'; 

Here I set lambda = 1, but the problem appears for any value of lambda, except when lambda = 0. When lambda = 0, the OLS and the ridge estimates coincide, as they should.

    lambda1 = 1;
    [m,n] = size(X); % Size of X

OLS estimator and MSE:

    b_ols = ((X')*X)^(-1)*((X')*y);
    yhat_ols = X*b_ols;
    MSE_ols = mean((y-yhat_ols).^2)

Ridge estimator and MSE:

    b_ridge = ((X')*X+lambda1*eye(n))^(-1)*((X')*y);
    yhat_ridge = X*b_ridge;
    MSE_ridge = mean((y-yhat_ridge).^2)

For the OLS regression, MSE = 0.0370 and for the ridge regression MSE = 0.1021.

$\endgroup$
  • $\begingroup$ It seems me that my answer is the only one, until now, that address your question properly. I would to know your opinion about it. $\endgroup$ – markowitz Sep 20 at 17:14
8
$\begingroup$

That is correct because $b_{OLS}$ is the minimizer of MSE by definition. The problem ($X^TX$ is invertible here) has only one minimum and any value other than $b_{OLS}$ will have higher MSE on the training dataset.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I am not sure if I understand your answer. In "The Elements of Statistical Learning" (Hastie et al. 2017 page 52) "The Gauss-Markov theorem implies that the least squares estimator has the smallest mean squared error of all linear estimators with no bias. However, there may well exist a biased estimator with smaller mean squared error." Isn't ridge an example of such a biased estimator? $\endgroup$ – Aristide Herve Sep 15 at 14:57
  • $\begingroup$ @AristideHerve Read my answer. $\endgroup$ – markowitz Sep 15 at 15:10
  • 4
    $\begingroup$ @AristideHerve The out of sample/test error can be lower but your case is the in-sample/training error. The linear regression formula you use minimizes $||Xw-y||$ and finds a $w$. So, basically, you shouldn't come up with a different $w$ and minimize the same cost, because this would contradict the first result. It's like minimising $f(x)=(x-2)^2$, finding that $x=2$ and trying out other values for a better result. $\endgroup$ – gunes Sep 15 at 15:27
  • $\begingroup$ @gunes, in practice RIDGE estimator are useful in prediction, then for out of sample performance. However, even for in sample only, I fear that what you said is correct, let me say, for “given data” but not for “given model”. In other word what you said is correct in general math terms but in the context of $MSE$ model specification matters and OLS is not necessarily the MSE minimizer. $\endgroup$ – markowitz Sep 15 at 15:38
  • 1
    $\begingroup$ Yeah, I'm not arguing about the benefit we get from regularisation in general, or the generalisation performance. The OP is trying to beat the training MSE obtained already by optimisation. $\endgroup$ – gunes Sep 15 at 15:44
1
$\begingroup$

like gunes said, the hastie quote applies to out-of-sample (test) MSE, whereas in your question you are showing us in-sample (training) MSE, which Hastie is not referring to.

For your in-sample case, maybe check mean absolute error instead, MAE, which will put the OLS and ridge on equal footing. Otherwise OLS has the upper hand if MSE is the performance criterion since it actively solves the plain MSE formula whereas ridge doesn't

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Why do you say that MAE puts the OLS and ridge on equal footing? $\endgroup$ – Dave Sep 15 at 19:40
  • $\begingroup$ there are the ridge predictions in a vector, and there are the OLS predictions in a vector. each differenced by the actual $y$ gives the prediction error. if we then applied the MSE formula to both prediction error vectors separately, we already know that OLS' objective function is the MSE so that makes it unfair for ridge which has a different objective (penalized MSE). If instead we apply the MAE formula to both prediction error vectors, neither ridge or OLS were programmed to solve for MAE, which is why they are on equal footing $\endgroup$ – develarist Sep 15 at 19:48
  • $\begingroup$ I see your point and offer the following counterargument: we (often) assume a Gaussian response variable, in which case, conditional mean and conditional median are equal (at the population level); therefore, while OLS was not trained explicitly to optimize MAE, it has a leg up. I would be interested in a simulation study of this, however, as this idea of mine might not pan out. $\endgroup$ – Dave Sep 15 at 19:54
  • $\begingroup$ when i said "maybe check" = "try out" $\endgroup$ – develarist Sep 15 at 19:55
  • $\begingroup$ Quick simulation of mine showed that it worked...how ‘bout that... $\endgroup$ – Dave Sep 16 at 12:23
1
$\begingroup$

Ordinary least squares (OLS) minimizes the residual sum of squares (RSS) $$ RSS=\sum_{i}\left( \varepsilon _{i}\right) ^{2}=\varepsilon ^{\prime }\varepsilon =\sum_{i}\left( y_{i}-\hat{y}_{i}\right) ^{2} $$

The mean squared deviation (in the version you are using it) equals $$ MSE=\frac{RSS}{n} $$ where $n$ is the number of observations. Since $n$ is a constant, minimizing the RSS is equivalent to minimizing the MSE. It is for this reason, that the Ridge-MSE cannot be smaller than the OLS-MSE. Ridge minimizes the RSS as well but under a constraint and as long $\lambda >0$, this constraint is binding. The answers of gunes and develarist already point in this direction.

As gunes said, your version of the MSE is the in-sample MSE. When we calculate the mean squared error of a Ridge regression, we usually mean a different MSE. We are typically interested in how well the Ridge estimator allows us to predict out-of-sample. It is here, where Ridge may for certain values of $\lambda $ outperform OLS.

We usually do not have out-of-sample observations so we split our sample into two parts.

  1. Training sample, which we use to estimate the coefficients, say $\hat{\beta}^{Training}$
  2. Test sample, which we use to assess our prediction $\hat{y}% _{i}^{Test}=X_{i}^{Test}\hat{\beta}^{Training}$

The test sample plays the role of the out-of-sample observations. The test-MSE is then given by $$ MSE_{Test}=\sum_{i}\left( y_{i}^{Test}-\hat{y}_{i}^{Test}\right) ^{2} $$

Your example is rather small, but it is still possible to illustrate the procedure.

% Generate Data. 
    X = [3, 3
        1.1 1
        -2.1 -2
        -2 -2]; 
    y =  [1 1 -1 -1]'; 
% Specify the size of the penalty factor
    lambda = 4;
% Initialize 
    MSE_Test_OLS_vector = zeros(1,m); 
    MSE_Test_Ridge_vector = zeros(1,m); 
% Looping over the m obserations
    for i = 1:m
        % Generate the training sample 
            X1 = X; X1(i,:) = [];
            y1 = y; y1(i,:) = [];
        % Generate the test sample
            x0 = X(i,:);        
            y0 = y(i); 
        % The OLS and the Ridge estimators
            b_OLS = ((X1')*X1)^(-1)*((X1')*y1);
            b_Ridge = ((X1')*X1+lambda*eye(n))^(-1)*((X1')*y1);
        % Prediction and MSEs
            yhat0_OLS = x0*b_OLS;
            yhat0_Ridge = x0*b_Ridge;
            mse_ols = sum((y0-yhat0_OLS).^2);
            mse_ridge = sum((y0-yhat0_Ridge).^2);
        % Collect Results
            MSE_Test_OLS_vector(i) = mse_ols;
            MSE_Test_Ridge_vector(i) = mse_ridge;
    end
% Mean MSEs
    MMSE_Test_OLS = mean(MSE_Test_OLS_vector)
    MMSE_Test_Ridge = mean(MSE_Test_Ridge_vector)
% Median MSEs
    MedMSE_Test_OLS = median(MSE_Test_OLS_vector)
    MedMSE_Test_Ridge = median(MSE_Test_Ridge_vector)

With $\lambda =4$, for example, Ridge outperforms OLS. We find the following median MSEs:

  • MedMSE_Test_OLS = 0.1418
  • MedMSE_Test_Ridge = 0.1123.

Interestingly, I could not find any value of $\lambda $ for which Ridge performs better when we use the average MSE rather than the median. This may be because the data set is rather small and single observations (outliers) may have a large bearing on the average. Maybe some others want to comment on this.

Regression Results The first two columns of the table above show the results of a regression of $x_{1}$ and $x_{2}$ on $y$ separately. Both coefficients positively correlate with $y$. The large and apparently erratic sign change in column 3 is a result of the high correlation of your regressors. It is probably quite intuitive that any prediction based on the erratic OLS estimates in column 3 will not be very reliable. Column 4 shows the result of a Ridge regression with $\lambda=4$.

Important note: Your data are already centered (have a mean of zero), which allowed us to ignore the constant term. Centering is crucial here if the data do not have a mean of zero, as you do not want the shrinkage to be applied to the constant term. In addition to centering, we usually normalize the data so that they have a standard deviation of one. Normalizing the data assures that your results do not depend on the units in which your data are measured. Only if your data are in the same units, as you may assume here to keep things simple, you may ignore the normalization.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Your answer follow the same spirit of those of Gunes and Deveralist. It seems me that these are wrong, or at least they address the problem in a wrong way. Those aspect emerge from my answer but let me say something more here. Firstly we have to focus on the definition of “mean square error”. It is a statistical object and it is not a sort of synonym of “quadratic form” in general math sense. From quadratic forms minimization, It is well known that, by construction, in regression analysis and given the regressors, minimize the $RSS$ more than in OLS case is not possible. $\endgroup$ – markowitz Sep 22 at 16:16
  • $\begingroup$ Now, estimators like RIDGE or LASSO are biased and following your definition of MSE are also less precise. Then we have to conclude that RIDGE or LASSO are useless estimators. This is not possible, infact your definition of MSE sound much more like variance of regression than MSE of it. Moreover in prediction terms, the $MSE$ minimization is a typical goal while your approach is equivalent to the $R^2$ maximization; a useless goal in regression. Infact $MSE$ can be focused on parameters or predicted values but in any case it address, in some way, the quality of the model. $\endgroup$ – markowitz Sep 22 at 16:17
  • $\begingroup$ In order to deal with quality of regression "specification matters"; bias term, then the true model, must appears. Moreover split the MSE in “in sample MSE” and “out of sample MSE” can be dangerous about the interpretation of MSE that, as pointed out in my answer and ref therein, is a population metrics. Infact in machine learning literature something like “in sample MSE” do not follow your definition, indeed it look like “training error”. In machine learning literature “in sample MSE” = “training error” + “optimism bias”. $\endgroup$ – markowitz Sep 22 at 16:17
1
$\begingroup$

As others have pointed out, the reason $β_{λ=0}$ (OLS) appears to have lower MSE than $β_{λ>0}$ (ridge) in your example is that you computed both values of $β$ from a matrix of four (more generally, $N$) observations of two (more generally, $P$) predictors $X$ and corresponding four response values $Y$ and then computed the loss on these same four observations. Forgetting OLS versus ridge for a moment, let's compute $β$ manually; specifically, we seek $β$ such that it minimizes the MSE of the in-sample data (the four observations). Given that $\hat{Y}=Xβ$, we need to express in-sample MSE in terms of $β$.

$MSE_{in-sample}=\frac{1}{N}\|Y-Xβ\|^2$

$MSE_{in-sample}=\frac{1}{N}[(Y-Xβ)^T(Y-Xβ)]$

$MSE_{in-sample}=\frac{1}{N}[Y^TY-2β^TX^TY+β^TX^TXβ]$

To find the value of $β$ minimizing this expression, we differentiate the expression with respect to $β$, set it equal to zero, and solve for $β$. I will omit the $\frac{1}{N}$ at this point since it's just a scalar and has no impact on the solution.

$\frac{d}{dβ}[Y^TY-2β^TX^TY+β^TX^TXβ]=0$

$-2X^TY+2X^TXβ=0$

$X^TXβ=X^TY$

$β=(X^TX)^{-1}X^TY$

Which is a familiar result. By construction, this is the value of $β$ that results in the minimum in-sample MSE. Let's generalize this to include a ridge penalty $λ$.

$β=(X^TX+λI)^{-1}X^TY$

Given the foregoing, it's clear that for $λ>0$, the in-sample MSE must be greater than that for $λ=0$.

Another way of looking at this is to consider the parameter space of $β$ explicitly. In your example there are two columns and hence three elements of $β$ (including the intercept):

$ \begin{bmatrix} β_0 \\ β_1 \\ β_2 \\ \end{bmatrix} $

Now let us further consider a point of which I will offer no proof (but of which proof is readily available elsewhere): linear models' optimization surfaces are convex, which means that there is only one minimum (i.e., there are no local minima). Hence, if the fitted values of parameters $β_0$, $β_1$, and $β_2$ minimize in-sample MSE, there can be no other set of these parameters' values with in-sample MSE equal to, or less than, the in-sample MSE associated with these values. Therefore, $β$ obtained by any process not mathematically equivalent to the one I walked through above will result in greater in-sample MSE. Since we found that in-sample MSE is minimized when $λ=0$, it is apparent that in-sample MSE must be greater than this minimum when $λ>0$.

$\Large{\text{A note on MSE estimators, in/out of sample, and populations:}}$

The usefulness of the ridge penalty emerges when predicting on out-of-sample data (values of the predictors $X$ on which the model was not trained, but for which the relationships identified in the in-sample data between the predictors and the response are expected to hold), where the expected MSE applies. There are numerous resources online that go into great detail on the relationship between $λ$ and the expected bias and variance, so in the interest of brevity (and my own laziness) I will not expand on that here. However, I will point out the following relationship:

$\hat{MSE}=\hat{bias}^2+\hat{var}$

This is the decomposition of the MSE estimator into its constituent bias and variance components. Within the context of linear models permitting a ridge penalty ($λ>=0$), it is generally the case that there is some nonzero value of $λ$ that results in its minimization. That is, the reduction (attributable to $λ$) in $\hat{var}$ eclipses the increase in $\hat{bias}^2$. This has absolutely nothing to do with the training of the model (the foregoing mathematical derivation) but rather has to do with estimating its performance on out-of-sample data. The "population," as some choose to call it, is the same as the out-of-sample data I reference because even though the "population" implicitly includes the in-sample data, the concept of a "population" suggests that infinite samples may be drawn from the underlying process (quantified by a distribution) and hence the influence of the in-sample data's idiosyncracies on the population vanish to insignificance.

Personally, after writing the foregoing paragraph, I'm even more sure that the discussion of "populations" adds needless complexity to this matter. Data were either used to train the model (in-sample) or they weren't (out-of-sample). If there's a scenario in which this distinction is impossible/impractical I've yet to see it.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I surprised from your answer. Indeed in a past comment you said: “I think that the hyper-syntactical focus on whether MSE should be seen as different for in/out of sample vs. as a population metric is inappropriate.” and I agree with this comment. Indeed if you read carefully my answer I do not defend this split as useful here. In the note that asker suggested the split was not considered. Moreover "in sample MSE=training MSE" is bad estimator for MSE; these fact put in bad light your story above. Story that move around and around the split in/out. $\endgroup$ – markowitz 9 hours ago
  • $\begingroup$ I have to repeat me agan: MSE is not a synonym for "mean square residual", in MSE loss, specification matters. So bias term must have chance to appear. It seems that in this discussion I’am the only person that address properly this crucial definitional issue. $\endgroup$ – markowitz 9 hours ago
-2
$\begingroup$

The result that gunes underscore, efficiency of OLS estimators, is valid among unbiased estimators. The RIDGE estimator induce bias in the estimates but can achieve lower MSE. See the start of the story that bring at the theorem that you cited (bias-variance tradeoff). At practical level RIDGE estimator is useful in prediction, mainly in context of big data (many predictors). In this context the out of sample performance of naive OLS regression are usually poorer than the RIDGE one.

uploading: the question/title is:

Mean squared error of OLS smaller than Ridge?

then, in order to contextualize and remove ambiguity, we have to consider not only his explanation but also the argument that Aristide Herve suggested in the comment to gunes (the first) answer (Gauss Markov theorem and another theorem [1.2 pag 15] in those lecture note [https://arxiv.org/pdf/1509.09169;Lecture]; unfortunately the link was deleted by he). My reply was based on those consideration.

The definition of MSE can be written on estimation of parameters or predicted values (https://en.wikipedia.org/wiki/Mean_squared_error) but from the above arguments the relevant here is that related to parameters. Then:

$MSE(\hat{\beta})=E[(\hat{\beta} - \beta)^2 ]$ given $\beta$ (true value)

note that at least in those definition the sample split train/test is not considered. All data are considered. Moreover the term $bias^2$ emerge.

Now from the lecture note we can check that for $\lambda>0$

$E[\hat{\beta}_{RIDGE}] \neq \beta$ then it is a biased estimator

$V[\hat{\beta}_{RIDGE}] < V[\hat{\beta}_{OLS}]$

and for some value of $\lambda>0$

$MSE[\hat{\beta}_{RIDGE}] < MSE[\hat{\beta}_{OLS}]$

infact we can read (pag 16):

Theorem 1.2 can also be used to conclude on the biasedness of the ridge regression estimator. The Gauss-Markov theorem (Rao, 1973) states (under some assumptions) that the ML regression estimator is the best linear unbiased estimator (BLUE) with the smallest MSE. As the ridge regression estimator is a linear estimator and outperforms (in terms ofMSE) this ML estimator, it must be biased (for it would otherwise refute the Gauss-Markov theorem).

for OLS the same consideration hold.

Therefore the reply of gunes

That is correct because $b_{OLS}$ is the minimizer of MSE by definition.

is wrong, and the consideration of develarist

like gunes said, the hastie quote applies to out-of-sample (test) MSE, whereas in your question you are showing us in-sample (training) MSE, which Hastie is not referring to.

is wrong too, the Gauss Markov theorem do not consider the sample split and the no bias condition is crucial there (https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem).

Therefore: Mean squared error of OLS smaller than Ridge? No, not always. It depends on the value of $\lambda$.

Now remain to say what went wrong in the computation of Aristide Herve. There are at least two problems. The first is that his suggestion are referred to $MSE$ in parameters estimation sense while your computation is focused on fitted/predicted values. In the last sense is usual to refers on Expected Prediction Error ($EPE$) and not on the Residual Sum of Square ($RSS$). Actually, for any linear model, is not possible to minimize $RSS$ more than OLS case. The explanation/comments of gunes sound like this and it is correct in this sense; however the minimization of $MSE$ is not the same thing. More important, in order to check the $MSE$ capability of several techniques or models in theoretical ground we have to consider the true model also, then to know the bias. Aristide Herve procedure do not consider this element, therefore cannot be adequate.

Finally we can also note that something like “in sample MSE” written on fitted values, that Dave, develarist and gunes refers on, have a dubious meaning. Infact in the spirit of $MSE$ we must to take into account the bias also, as I already said specification matters, while if we are focused only on residuals (in sample errors) it cannot emerge. Worse, regardless the linearity of the estimated model is always possible to achieve a perfect in sample fit, then to achieve “in sample MSE=0”. This discussion give us the last clarifications: Is MSE decreasing with increasing number of explanatory variables?

infact Cagdas Ozgenc show there that $MSE$ should be intended as population metrics and

$E[\hat{MSE_{in}}]<MSE$ (downward biased, after all this is obvious)

while $E[\hat{MSE_{out}}]=MSE$

therefore $\hat{MSE_{in}}$ is not what we need. This conclude the story.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Yes, thanks. I expected the ridge estimator to achieve a lower MSE at least for some values of lambda. But in the example, the ridge-MSE is higher for all values of lambda (except for lambda = 0, when both MSEs are equal). This is what confuses me. $\endgroup$ – Aristide Herve Sep 15 at 15:15
  • $\begingroup$ -1 OLS doesn’t have the lowest square loss among unbiased estimators; OLS has the lowest square loss among all estimators. The “LS” means “least squares” as in “lowest square loss”. You’re correct, however, that a ridge regression could have stronger out-of-sample performance than an OLS. $\endgroup$ – Dave Sep 16 at 12:16
  • $\begingroup$ I’m not sure about your point. It is related to my comment about gunes answer. I known what LS mean, if the data are given and we are interested in minimization of a related quadratic form, you are right. However in MSE/statistic point of view, we have to add something to the story. We have to start from a given true model and, so, it seem me that specification problems matters. Today I do not have time but more ahead maybe I add something to my answer. $\endgroup$ – markowitz Sep 16 at 12:47
  • $\begingroup$ However just now you can consider the Gauss Markov theorem as reported by Aristide Herve in the comment about gunes answer, and theorem 1.2 at pag 15 here arxiv.org/pdf/1509.09169;Lecture. Both result consider the bias problem and It seem that both result hold regardless the split in/out of sample. Me too have to check better, but if I right I ask you to upvote. $\endgroup$ – markowitz Sep 16 at 12:47
  • 2
    $\begingroup$ Ridge cannot achieve a lower in-sample MSE. $\endgroup$ – Dave Sep 16 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.