The figure below shows the Test-MSE against $\lambda $, the penalty term. There are two minima, one very close to 0 and the other at around 7.
These are made-up data I wanted to use in an introductory class about the Ridge regression. I did not expect to find two minima. My questions:
- Do multiple minima often arise in practice?
- Is there a standard way to discard one of the minima? (Stata, for example, only reports the larger of the two minima.)
- There seems to be a theoretical result that states that the ridge loss function is strictly convex (on page 17, https://arxiv.org/pdf/1509.09169;Lecture refers to a theorem by Fletcher (2008) and states that the ridge estimator is a global minimum). What assumptions are violated in this example so that the theorem does not apply?
Related question: Can cross validation MSE have multiple minima as function of lambda?
Below the data and Matlab code. Note: this is not meant to be good programming. The students have no experience with programming.
% Generate data
X = [3, 3
1.1 .9
-2.1 -1.9
-2 -2];
y = [1 1 -1 -1]';
[n,p] = size(X);
%% Partition data into 4 folds (with four observations, this corresponds to LOO)
K = 4;
cv = cvpartition(numel(y), 'kfold',K);
%% Loop over lambda
j = 1;
for lambda = 0:0.01:12
mse_OLS = zeros(K,1);
for k=1:K
% training/testing indices for this fold
trainIdx = cv.training(k);
testIdx = cv.test(k);
% train Ridge
pseudo = sqrt(lambda) * eye(p);
Zplus = [X(trainIdx,:);pseudo];
yplus = [y(trainIdx);zeros(p,1)];
b_Ridge = Zplus\yplus;
% compute mean squared error
mse_Ridge(k) = mean((y(testIdx) - X(testIdx,:)*b_Ridge).^2);
end
% average RMSE across k-folds
lambda_vector(:,j) = lambda;
b_Ridge_vector(:,j) = ((((X')*X + lambda*eye(p)))^(-1))*(X')*y;
avrg_rmse_Ridge(:,j) = mean(sqrt(mse_Ridge));
j = j+1;
end
[M,I] = min(avrg_rmse_Ridge)
lambda_opt = lambda_vector(I)
end of code