2
$\begingroup$

Until now, I treated the maximal likelihood method to fit a logistic regression model always as a black box. But recently, in the paper "Categorization Based on Regularized Linear Classification methods" by T. Zhang, F.J. Oles, on on pp 10, they describe a setting that is not known to me.

Namely, they begin by considering the conditional probability $\mathbb{P}(Y=1|w,x)$. But what does "$w$" even mean ?

Earlier in the paper they only say that $w$ is a weight vector. Also, this notation "$w,x$"is not known to me. What does that mean ?

They go then on to say that they want to estimate $w$ by the maximum likelihood which minimizes

$$\hat w = \arg \inf_w (\text{"complicated function of $w,x_i,y_i$"}).$$

I assume I can make sense of this minimization only after I have understood the answers to the previous questions.

$\endgroup$
5
$\begingroup$

In general, $P(A|B)$ is read as "probability of $A$ given $B$" or in other words, the probability of event $A$ under the assumption of $B$. If there are multiple events that we assume are given, we can simply write $P(A|B,C)$, which mean that we now assume $B$ and $C$.

From the paper:

In logistic regression, we model the conditional probability $P(y = 1 | w, x)$ as $1/(\exp(−w^T x) + 1)$.

So in this case, the model $P(y=1|w,x)$ is the probability that $y=1$ given the weight vector $w$ and the input vector $x$. Throughout the literature, this is sometimes written as $P(y=1|x)$, where the $w$ is left implicit. It is usually intended to be clear by context that the parameters of the model (in this case, the weights $w$) are part of the conditional probability.

Our goal is to come up with a weight vector $\hat{w}$ that maximizes $P(y=1|\hat{w},x)$, so that when we receive a test input $x'$, we can plug it into our model along with the weights $\hat{w}$. If the true value of $x'$ is $y'=1$, we want $P(y'=1|\hat{w},x')$ to be high. Note that we can apply a natural log to the model to obtain $1-\ln(\exp(-w^Tx)+1)$, which will be useful in formulating an optimization problem to learn $w$. With a little work, we obtain the expression $$ \hat{w}=\operatorname{arg}\inf_w\frac{1}{n}\sum_{i=1}^n\ln(\exp(-w^Tx_iy_i)+1) $$ which gives us a way to obtain the weight vector $\hat{w}$ which maximizes the conditional probability using a collection of labeled data $\{(x_i,y_i)\}_{i=1}^n$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 Could you maybe please also give me a hint how to this approach to logistic regression relates to the usual one, where no $w$ is present ? The one I knew was in the book "An Introduction to Statistical Learning with Applications in R" by James G., Witten D., Hastie T., Tibshirani R. on pp. 131, where no $w$ was. And I have to understand how the estimate $\hat w$ relates to the standard logistic regression, as described in that book. $\endgroup$ – l7ll7 Nov 15 '16 at 15:01
  • $\begingroup$ I updated my answer with a couple of sentences after the block quote. In short, the parameters (weights $w$, in this case) are left out of the conditional for convenience. We can define $w=(\beta_0,\beta_1)$ and let $x=(1,x)$ to obtain the expressions in the book, which appears to discuss logistic regression for a scalar $x$. The problem of estimating $(\beta_0,\beta_1)$ is exactly the same as the problem of estimating $w$. Read carefully, and I'm sure you'll be able to see this. The paper you linked above is a more general case where $w$ and $x$ are vectors. $\endgroup$ – scherm Nov 15 '16 at 15:10
0
$\begingroup$

For illustration, suppose that there are n elements($x_1$ to $x_i$) determining whether or not a man gets cancer($y=0$ or $y=1$). We have some data collected before as examples. For instance $X_j$ when $x_1 = 1, x_2 = 4.1$, .. y = 1 and $X_{j+1}$ when $x_1 = 0, x_2 = 1.2$, .. and etc then $y = 0$.

We modify the linear regression a little bit. In linear regression, the output of $\sum_{i=0}^n \lambda_i * x_i$ is compared to the label y(using the least squared error), while in logistic regression we change the output to a probability using the logistic function $\frac{1}{1+e^X}$, $\frac{1}{1+e^{-(\lambda_0 + \sum_{i=1}^n\lambda_i x_i)}}$. What we go to optimize is the product of all the probabilities of all the examples(as we did for the normal distribution) if the examples are independent to each other by adjusting $\lambda_i$(as we did for $\mu$ and $\sigma$ in normal distribution). This probability is noted as the conditional probability $p(Y=1|X;\lambda)$(note that it is only for Y=1, and the $\lambda$ is the estimator vector $(\lambda_0, \lambda_1,..., \lambda_n)$ and the conditional likelihood is $l(\lambda; Y=1|X)$. We use the former after we have trained the estimator using the given examples(the probability may between 0 and 1, we can adjust all probability larger than 0.5 as 1 otherwise 0).

So, just like that in linear regression, $w$ is the estimator(or say constraints or weights) to be trained and $x$ is the training data(in precise the independent variable).

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.