What does the “$w,x$” from $\mathbb{P}(Y=1|w,x)$ mean?

Question

Until now, I treated the maximal likelihood method to fit a logistic regression model always as a black box. But recently, in the paper "Categorization Based on Regularized Linear Classification methods" by T. Zhang, F.J. Oles, on on pp 10, they describe a setting that is not known to me.

Namely, they begin by considering the conditional probability $\mathbb{P}(Y=1|w,x)$. But what does "$w$" even mean ?

Earlier in the paper they only say that $w$ is a weight vector. Also, this notation "$w,x$"is not known to me. What does that mean ?

They go then on to say that they want to estimate $w$ by the maximum likelihood which minimizes

$$\hat w = \arg \inf_w (\text{"complicated function of $w,x_i,y_i$"}).$$

I assume I can make sense of this minimization only after I have understood the answers to the previous questions.

Answer 1

In general, $P(A|B)$ is read as "probability of $A$ given $B$" or in other words, the probability of event $A$ under the assumption of $B$. If there are multiple events that we assume are given, we can simply write $P(A|B,C)$, which mean that we now assume $B$ and $C$.

From the paper:

In logistic regression, we model the conditional probability $P(y = 1 | w, x)$ as $1/(\exp(−w^T x) + 1)$.

So in this case, the model $P(y=1|w,x)$ is the probability that $y=1$ given the weight vector $w$ and the input vector $x$. Throughout the literature, this is sometimes written as $P(y=1|x)$, where the $w$ is left implicit. It is usually intended to be clear by context that the parameters of the model (in this case, the weights $w$) are part of the conditional probability.

Our goal is to come up with a weight vector $\hat{w}$ that maximizes $P(y=1|\hat{w},x)$, so that when we receive a test input $x'$, we can plug it into our model along with the weights $\hat{w}$. If the true value of $x'$ is $y'=1$, we want $P(y'=1|\hat{w},x')$ to be high. Note that we can apply a natural log to the model to obtain $1-\ln(\exp(-w^Tx)+1)$, which will be useful in formulating an optimization problem to learn $w$. With a little work, we obtain the expression $$ \hat{w}=\operatorname{arg}\inf_w\frac{1}{n}\sum_{i=1}^n\ln(\exp(-w^Tx_iy_i)+1) $$ which gives us a way to obtain the weight vector $\hat{w}$ which maximizes the conditional probability using a collection of labeled data $\{(x_i,y_i)\}_{i=1}^n$.

Answer 2

For illustration, suppose that there are n elements($x_1$ to $x_i$) determining whether or not a man gets cancer($y=0$ or $y=1$). We have some data collected before as examples. For instance $X_j$ when $x_1 = 1, x_2 = 4.1$, .. y = 1 and $X_{j+1}$ when $x_1 = 0, x_2 = 1.2$, .. and etc then $y = 0$.

We modify the linear regression a little bit. In linear regression, the output of $\sum_{i=0}^n \lambda_i * x_i$ is compared to the label y(using the least squared error), while in logistic regression we change the output to a probability using the logistic function $\frac{1}{1+e^X}$, $\frac{1}{1+e^{-(\lambda_0 + \sum_{i=1}^n\lambda_i x_i)}}$. What we go to optimize is the product of all the probabilities of all the examples(as we did for the normal distribution) if the examples are independent to each other by adjusting $\lambda_i$(as we did for $\mu$ and $\sigma$ in normal distribution). This probability is noted as the conditional probability $p(Y=1|X;\lambda)$(note that it is only for Y=1, and the $\lambda$ is the estimator vector $(\lambda_0, \lambda_1,..., \lambda_n)$ and the conditional likelihood is $l(\lambda; Y=1|X)$. We use the former after we have trained the estimator using the given examples(the probability may between 0 and 1, we can adjust all probability larger than 0.5 as 1 otherwise 0).

So, just like that in linear regression, $w$ is the estimator(or say constraints or weights) to be trained and $x$ is the training data(in precise the independent variable).