I'm building a logistic regression model to predict if a patient has cancer based on 9 features. Having built learning curves and increasing my regularization parameter to reduce over fitting, which caused my accuracy to increase from 82% to 86%, I'm stuck on options of how to improve it further. I'm using a training set of 69 samples, a validation set of 24 samples and a test set of 24 samples. (60%,20%,20% split) (I'm doing this on octave)

When I drew my initial learning curve, I saw that the validation error is actually lower than the training error. Then according to https://stats.stackexchange.com/questions/187335/validation-error-less-than-training-error/187404#187404 , as the issue lies in overfitting, I increased the regularization parameter (lambda), which caused the learning curve to show characteristics of high bias, and the accuracy on the Test Set increased to 86%.

Learning curve code which causes validation error to be less than training error;

%costlogreg is a function to calculate logistic regression cost
%linearRegcost is a function to calculate the training cost and validation cost
%sigmoid2 is just a simple logistic function
%listJ is a matrix of zeros 
listJ = zeros(69,3);

for i = 1:69

initial_theta = zeros(size(Xadjust,2) + 1,1);
%Xtrain, Ytrain is the training set
Xtrain = Xadjust([1:i],:);
Ytrain = Y([1:i],:);

%function handle for costlogreg
costfunction = @(t)costlogreg(t,Xtrain,Ytrain,lambda);

%obtains theta depending on training set
options = optimset('Gradobj','on','MaxIter',50);
[theta] = fmincg(costfunction, initial_theta,options);

%calculates the cost on the training set and adds it to a list
[Jtrain,~] = linearRegCost(theta,Xtrain,Ytrain);
listJ(i,1) = i;
listJ(i,2) = Jtrain;

%obtains Jcv using earlier found theta
[Jval,~] = linearRegCost(theta,Xval,Yval);
listJ(i,3) = Jval;


Code to choose the best lambda value which gives higher accuracy on test set

%lambda set is a set of regularization parameters to test
%listlambda is a matrix of zeros to collect the accuracy on test set for each lambda used
listlambda = zeros(size(lambdaset,2),4);

for i = 1:size(lambdaset,2)

initial_theta = zeros(size(Xadjust,2) + 1,1);
costfunction = @(t)costlogreg(t,Xtrain,Ytrain,lambdaset(1,i));
options = optimset('Gradobj','on','MaxIter',50);
[theta] = fmincg(costfunction, initial_theta,options);
[Jtrain,~] = linearRegCost(theta,Xtrain,Ytrain);
listlambda(i,1) = lambdaset(1,i);
listlambda(i,2) = Jtrain;
[Jval,~] = linearRegCost(theta,Xval,Yval);
listlambda(i,3) = Jval;
[h] = sigmoid2(Xtest,theta);
list = zeros(size(h,1),1);
idx = find(h >= 0.5);
list(idx) = 1;
accuracy = mean(double(list == Ytest));
listlambda(i,4) = accuracy;

%figures the row with the highest accuracy on the test set
[~,maxidx] = (max(listlambda(:,4)));
bestlambda = listlambda(maxidx,1);

enter image description here