0
$\begingroup$

I am training the ridge regression on a one-day sensor data using the closed-form solution where $$ \beta=(X^TX+\lambda*I)^{-1}X^TY $$ and Matlab.

The $X$ is 15 polynomial time matrix. I created a loop where different lambdas were used in the ridge regression and then apply this model to the training set, and then computing the RMSE of the predicted results using the $\beta$ obtained from the equation above.

With lambda increasing, the RMSE decreases for smaller lambda value and then increases later, which is not expected. I would expect that the RMSE maintains increasing trend with increasing lambda value as the constrain increases. Just to clarify, I understand that the RMSE is expected to be decrease at first and then increase for a test set. However, this issue happens when I test the model only for the training data set.

I dont understand what causes this issue. Could anyone please help me with this? Much appreciated.

Note:

  1. I realized that my time array was not normalized, and this issue only occurs when the time array was scaled by random scaling factor. But I dont see why the scaling will cause this issue. PS. I have normalized the x matrix before the ridge regression. The RMSE increases with increasing lambda. However, the gradient of the RMSE is decreasing with increasing lambda, which is different than the one without normalization. Is there anyone with advance mathmethics background who is kind to provide any comments onto this? Which gradient change makes more sense to you?

enter image description here

  1. The plot of RSE with lambda is shown below: enter image description here

  2. Updated Notes: As per the comment from Edm below, it might due to the machine numerical error due to the large x value. Therefore, I computed the plots of RMSE vs lambda with different polynomial degree. The plot is shown below. The issue occurrs around polynomial degree around 15.Also, the largest value in the x matrix for 30 degree of polynomial is 2e^103. Based on my knowlegde, matlab can handle much larger number than this, up to around 1e^1023? Any comments regarding this is appreciated! enter image description here

  3. Apologize that I cant provide the dataset. But below is my matlab code if that helps?

%load the data
load xxx.mat

pm2d5= data.pm2d5;
time = data.time;
warning ('off','all');

fig = figure('Position', [0,0,850,1100]);
t_num=[1:59:60*5-1];%scaling factor

for iii=1:length(t_num)
%if change to seconds unit, times another 60, the results change.
time_num = (datenum(time)-floor(datenum(time)))*24*t_num(iii); %time to seconds and applied by the scaling factor


%set the xmatrix, 15 degree polynomial 
dim=15;
x_matrix=ones(length(time_num),dim+1);
for i=1:dim+1
    x_matrix(:,i)=time_num.^(dim+1-i);
end
dim=size(x_matrix,2);

%compute the RSE for Ridge method different lambda
num=0:0.001:3;
for ii=1:length(num)
lambda=num(ii);
theta=((x_matrix.'*x_matrix)+lambda*eye(dim))\(x_matrix.')*pm2d5;
yfit=x_matrix(:,1:end-1)*theta(1:end-1);
intercept=theta(end);
RSE(ii,iii)=sqrt(mse(yfit+intercept,pm2d5));
end
plot(num, RSE(:,iii),'DisplayName',num2str(t_num(iii)))
hold on 
end
legend
xlabel('lambda')
ylabel('RSE')
$\endgroup$
1
  • $\begingroup$ Increases beyond the RMSE given by the OLS regression? $\endgroup$
    – Dave
    Commented May 8 at 13:58

1 Answer 1

0
$\begingroup$

This is to be expected if these results are on a test set. See, for example, Figure 6.12 of An Introduction to Statistical Learning (ISL) by James et al.

Ridge regression makes a bias versus variance tradeoff (see Section 2.2.2 of ISL, or many pages on this site, about the tradeoff). An unpenalized linear regression model might be unbiased, but with a high ratio of predictors to observations there could be very high variance/MSE on a test set. Increasing the penalization increases the bias but can decrease the variance enough to lower the MSE. Too much penalization, however, will again lead to increased MSE.

Such results on the training set are difficult to understand. The unpenalized model should have the minimum root-mean-square error (RMSE) provided that there are fewer predictors than observations. As the dip in RMSE is relatively small, I suppose there might be some problem associated with machine precision when fitting a 15-degree polynomial with what seem to be numerically large x values.

A final comment: trying to fit a 15-degree polynomial is probably not a good idea in any event. Look into other ways of fitting time-course data flexibly, for example with cubic regression splines.

$\endgroup$
3
  • $\begingroup$ Thanks for the comment. I understand that the MSE will decrease first and increase later for a test set. However, I am undertaking this whole process only in the training set, not in the test set. I would expect for the training set only, the MSE would increase with larger penalty? $\endgroup$
    – christine
    Commented Nov 24, 2022 at 21:23
  • $\begingroup$ I agree that it is strange for the RMSE to decrease at first with increasing $\lambda$ on the training set. For $\lambda=0$, we should have the minimum (R)MSE OLS solution. $\endgroup$ Commented Nov 24, 2022 at 21:58
  • $\begingroup$ I also suspect that this issue is likely due to the large x number. I have computed the RMSE vs lambda plots with different degree of polynomial, which will be updated in my post above shortly. The plot shows that the issue is serious around polynomial of 15~16. I am not sure why this issue occurs at this particular range? Also, I understand that 15-degree polynomial is not good choice. I am only doing this for a course homeowork, which we are doing to explore the downside of overfitting. $\endgroup$
    – christine
    Commented Nov 24, 2022 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.