2
$\begingroup$

The figure below shows the Test-MSE against $\lambda $, the penalty term. There are two minima, one very close to 0 and the other at around 7.

MSE against Penalty

These are made-up data I wanted to use in an introductory class about the Ridge regression. I did not expect to find two minima. My questions:

  1. Do multiple minima often arise in practice?
  2. Is there a standard way to discard one of the minima? (Stata, for example, only reports the larger of the two minima.)
  3. There seems to be a theoretical result that states that the ridge loss function is strictly convex (on page 17, https://arxiv.org/pdf/1509.09169;Lecture refers to a theorem by Fletcher (2008) and states that the ridge estimator is a global minimum). What assumptions are violated in this example so that the theorem does not apply?

Related question: Can cross validation MSE have multiple minima as function of lambda?

Below the data and Matlab code. Note: this is not meant to be good programming. The students have no experience with programming.

% Generate data 
X = [3, 3
    1.1 .9
    -2.1 -1.9
    -2 -2]; 
y =  [1 1 -1 -1]'; 
[n,p] = size(X); 
%% Partition data into 4 folds (with four observations, this corresponds to LOO)
K = 4;
cv = cvpartition(numel(y), 'kfold',K);
%% Loop over lambda
j = 1; 
for lambda = 0:0.01:12
    mse_OLS = zeros(K,1);
    for k=1:K
        % training/testing indices for this fold
        trainIdx = cv.training(k);
        testIdx = cv.test(k);

        % train Ridge
            pseudo = sqrt(lambda) * eye(p);
            Zplus  = [X(trainIdx,:);pseudo];
            yplus  = [y(trainIdx);zeros(p,1)];
            b_Ridge = Zplus\yplus;

        % compute mean squared error
        mse_Ridge(k) = mean((y(testIdx) - X(testIdx,:)*b_Ridge).^2);
    end
    % average RMSE across k-folds
        lambda_vector(:,j) = lambda;
        b_Ridge_vector(:,j) = ((((X')*X + lambda*eye(p)))^(-1))*(X')*y;
        avrg_rmse_Ridge(:,j) = mean(sqrt(mse_Ridge));
    j = j+1;
end
[M,I] = min(avrg_rmse_Ridge)
lambda_opt = lambda_vector(I)

end of code

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.