1
$\begingroup$

According to what i learned in machine learning, the loss function is derived by the Maximum likelihood estimation of training data. Taking logistic regression as an example:
we got a train data set $\{x^{(i)}, y^{(i)}\}(i=1,..n)$, and assume the probability $y$ and the feature $x$ satisfy the formula $y = h(\theta^Tx) =\frac{1}{1+e^{-\theta^Tx}}$.
then we have the log likehood function on train data:
$ln(L(\theta;x,y)) = \sum_{i=1}^{n}y^{(i)}lnh(\theta^Tx^{(i)}) + (1-y^{(i)})ln(1-h(\theta^Tx^{(i)}))$
and the loss is the negative log likehood function.
$l(\theta) = \sum_{i=1}^{n}-y^{(i)}lnh(\theta^Tx^{(i)}) - (1-y^{(i)})ln(1-h(\theta^Tx^{(i)}))$
when i learned the weighted logistics regression, the loss function was given below:
$l(\theta) = \sum_{i=1}^{n}-w_1y^{(i)}lnh(\theta^Tx^{(i)}) - w_0(1-y^{(i)})ln(1-h(\theta^Tx^{(i)}))$
the $w_1$ represent the weight on the positive sample and $w_0$ represent the weight on the negative sample.(of course you can make every single sample a specific weight but here we take the simple assumption)
here comes my questions: why in weighted logistic regression the loss functions changes but the objective function keep the same as object function in logistic regression? in my opinion the loss function is derived by the likehood function and the likehood function is derived by the objective function, so the the objective function and the loss function are connected, it should not happen that one change but another remains.
thanks for any reply!

$\endgroup$
1
$\begingroup$

The objective in logistic regression is to maximize likelihood of data which is assumed to be Bernaulli-distributed. $$L(\theta)=\prod_i p_i^{y_i} (1-p_i)^{1-y_i}$$ where $p_i$ is given by the logistic function $g(z) = \frac 1 {1+e^{-z}}$, $z=\theta^Tx$.

Taking negative log of this quantity gives the loss function as you have mentioned.

Class-weighted logistic regression assigns $w_+$ weights to positive samples and $w_-$ weights to negative samples. But let us assume the general case where all samples have a weight $w_i$. In terms of likelihood this means each sample is now given a probability of occurence (as opposed to other samples) as $w_i$. (If $\sum_i w_i \neq 1$ then $w_i\leftarrow\frac{w_i}{\sum_i w_i}$). The likelihood of each sample is exponentiated by this probability.

So the likelihodd of all samples together becomes:

$$L(\theta)=\prod_i (p_i^{y_i} (1-p_i)^{1-y_i})^{w_i}$$

You can see how taking the negative log of this would give us the loss function for weighted logistic regression: $$J(\theta) = -\sum_i w_i [y_i \ln(p_i) + (1-y_i)\ln(1-p_i)]$$

where $p_i$ is the same as unweighted scenario.

Class weighted logistic regression basically says that $w_i$ is $w_+$ if $i^{th}$ sample is positive else $w_-$. It is trivial to see that this will indeed lead to the class-weighted logistic regression loss function as you mention.

$\endgroup$
1
  • $\begingroup$ i get it. the weight change the likehood function and the objective function remains. thank you so much! $\endgroup$ – ConnellyM Sep 28 '20 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.