I'm building a logistic regression model to predict if a patient has cancer based on 9 features. Having built learning curves and increasing my regularization parameter to reduce over fitting, which caused my accuracy to increase from 82% to 86%, I'm stuck on options of how to improve it further. I'm using a training set of 69 samples, a validation set of 24 samples and a test set of 24 samples. (60%,20%,20% split) (I'm doing this on octave)
When I drew my initial learning curve, I saw that the validation error is actually lower than the training error. Then according to https://stats.stackexchange.com/questions/187335/validation-error-less-than-training-error/187404#187404 , as the issue lies in overfitting, I increased the regularization parameter (lambda), which caused the learning curve to show characteristics of high bias, and the accuracy on the Test Set increased to 86%.
Learning curve code which causes validation error to be less than training error;
%costlogreg is a function to calculate logistic regression cost %linearRegcost is a function to calculate the training cost and validation cost %sigmoid2 is just a simple logistic function %listJ is a matrix of zeros listJ = zeros(69,3); for i = 1:69 initial_theta = zeros(size(Xadjust,2) + 1,1); %Xtrain, Ytrain is the training set Xtrain = Xadjust([1:i],:); Ytrain = Y([1:i],:); %function handle for costlogreg costfunction = @(t)costlogreg(t,Xtrain,Ytrain,lambda); %obtains theta depending on training set options = optimset('Gradobj','on','MaxIter',50); [theta] = fmincg(costfunction, initial_theta,options); %calculates the cost on the training set and adds it to a list [Jtrain,~] = linearRegCost(theta,Xtrain,Ytrain); listJ(i,1) = i; listJ(i,2) = Jtrain; %obtains Jcv using earlier found theta [Jval,~] = linearRegCost(theta,Xval,Yval); listJ(i,3) = Jval; end
Code to choose the best lambda value which gives higher accuracy on test set
%lambda set is a set of regularization parameters to test %listlambda is a matrix of zeros to collect the accuracy on test set for each lambda used listlambda = zeros(size(lambdaset,2),4); for i = 1:size(lambdaset,2) initial_theta = zeros(size(Xadjust,2) + 1,1); costfunction = @(t)costlogreg(t,Xtrain,Ytrain,lambdaset(1,i)); options = optimset('Gradobj','on','MaxIter',50); [theta] = fmincg(costfunction, initial_theta,options); [Jtrain,~] = linearRegCost(theta,Xtrain,Ytrain); listlambda(i,1) = lambdaset(1,i); listlambda(i,2) = Jtrain; [Jval,~] = linearRegCost(theta,Xval,Yval); listlambda(i,3) = Jval; [h] = sigmoid2(Xtest,theta); list = zeros(size(h,1),1); idx = find(h >= 0.5); list(idx) = 1; accuracy = mean(double(list == Ytest)); listlambda(i,4) = accuracy; endfor %figures the row with the highest accuracy on the test set [~,maxidx] = (max(listlambda(:,4))); bestlambda = listlambda(maxidx,1);