5
$\begingroup$

I'm trying to implement a logistic regression function in matlab. I calculated the theta values, linear regression cost function is converging and then I use those parameters in logistic regression function as a decision boundary.

I'm trying to obtain an overfit logistic regression tree to show how cost function behaves during overfitting with respect to training set size. I get a converging training error which is expected. I also get an increasing validation error which is also expected but even though they are increasing, my validation error values are always below the training error values. Is it suppossed to be like that (because we are using the logarithm of the hypothesis function and not the squared error function) or am I doing something wrong?. (I can't post images because I'm a new member)

My cost function calculations are included in the code part, I'd appreciate any help.

    for i=1:row
    if y(i)>0
        classification(i)=1;
    else
        classification(i)=0;
    end
classpredict(i)=1/(1+exp(-(ypredict(i))));
end

for i=1:size(data,1)-row
    if validy(i)>0
        classvalid(i)=1;
    else
        classvalid(i)=0;
    end
classvalidpredict(i)=1/(1+exp(-(validpredicty(i))));
end


for i=1:size(classpredict)

        costclasstrain=-(costclasstrain+(classification(i))*(log(classpredict(i)))+(1-classification(i))*(log(1-classpredict(i))))/row;
end

for i=1:size(classvalidpredict)

      costclassvalid=-(costclassvalid+(classvalid(i))*(log(classvalidpredict(i)))+(1-classvalid(i))*(log(1-classvalidpredict(i))))/(size(data,1)-row);
end
$\endgroup$
  • 2
    $\begingroup$ How much time have you spent reading texts and papers on the logistic regression model? $\endgroup$ – Frank Harrell Jul 27 '17 at 16:50
  • 1
    $\begingroup$ Why are you flipping the sign of the cost in every iteration? Shouldn't the minus sign be inside the parentheses? $\endgroup$ – Moss Murderer Jan 17 '18 at 7:39
1
$\begingroup$

In matlab or R I would suggest to use vectorized implementation, which logistic loss is just

$$ \mathbf 1^T (-b \log p -(1-b) \log(1-p)) $$

Where

$$p=\frac 1 {1+\exp{(-Ax)}}$$

In above notation, $A$ is data matrix and $b$ is the binary response.

Here is the matlab code (with gradient calculation)

function [J grad] = lg_loss(A,b,x)
    sigmoid = @(z) 1.0 ./ (1.0 + exp(-z));
    p=sigmoid(A*x);
    J=sum(-b.*log(p)-(1-b).*log(1-p));
    grad=A'*(p-b);
end
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.