# Ridge regression, large lambda results in smaller RMSE of the training data

I am training the ridge regression on a one-day sensor data using the closed-form solution where $$\beta=(X^TX+\lambda*I)^{-1}X^TY$$ and Matlab.

The $$X$$ is 15 polynomial time matrix. I created a loop where different lambdas were used in the ridge regression and then apply this model to the training set, and then computing the RMSE of the predicted results using the $$\beta$$ obtained from the equation above.

With lambda increasing, the RMSE decreases for smaller lambda value and then increases later, which is not expected. I would expect that the RMSE maintains increasing trend with increasing lambda value as the constrain increases. Just to clarify, I understand that the RMSE is expected to be decrease at first and then increase for a test set. However, this issue happens when I test the model only for the training data set.

I dont understand what causes this issue. Could anyone please help me with this? Much appreciated.

Note:

1. I realized that my time array was not normalized, and this issue only occurs when the time array was scaled by random scaling factor. But I dont see why the scaling will cause this issue. PS. I have normalized the x matrix before the ridge regression. The RMSE increases with increasing lambda. However, the gradient of the RMSE is decreasing with increasing lambda, which is different than the one without normalization. Is there anyone with advance mathmethics background who is kind to provide any comments onto this? Which gradient change makes more sense to you?

1. The plot of RSE with lambda is shown below:

2. Updated Notes: As per the comment from Edm below, it might due to the machine numerical error due to the large x value. Therefore, I computed the plots of RMSE vs lambda with different polynomial degree. The plot is shown below. The issue occurrs around polynomial degree around 15.Also, the largest value in the x matrix for 30 degree of polynomial is 2e^103. Based on my knowlegde, matlab can handle much larger number than this, up to around 1e^1023? Any comments regarding this is appreciated!

3. Apologize that I cant provide the dataset. But below is my matlab code if that helps?

pm2d5= data.pm2d5;
time = data.time;
warning ('off','all');

fig = figure('Position', [0,0,850,1100]);
t_num=[1:59:60*5-1];%scaling factor

for iii=1:length(t_num)
%if change to seconds unit, times another 60, the results change.
time_num = (datenum(time)-floor(datenum(time)))*24*t_num(iii); %time to seconds and applied by the scaling factor

%set the xmatrix, 15 degree polynomial
dim=15;
x_matrix=ones(length(time_num),dim+1);
for i=1:dim+1
x_matrix(:,i)=time_num.^(dim+1-i);
end
dim=size(x_matrix,2);

%compute the RSE for Ridge method different lambda
num=0:0.001:3;
for ii=1:length(num)
lambda=num(ii);
theta=((x_matrix.'*x_matrix)+lambda*eye(dim))\(x_matrix.')*pm2d5;
yfit=x_matrix(:,1:end-1)*theta(1:end-1);
intercept=theta(end);
RSE(ii,iii)=sqrt(mse(yfit+intercept,pm2d5));
end
plot(num, RSE(:,iii),'DisplayName',num2str(t_num(iii)))
hold on
end
legend
xlabel('lambda')
ylabel('RSE')
• Increases beyond the RMSE given by the OLS regression?
– Dave
Commented May 8 at 13:58

• I agree that it is strange for the RMSE to decrease at first with increasing $\lambda$ on the training set. For $\lambda=0$, we should have the minimum (R)MSE OLS solution. Commented Nov 24, 2022 at 21:58