0
$\begingroup$

I was reading a popular article on adversarial training. https://adversarial-ml-tutorial.org/linear_models/

It says, In this case, rather than use multi-class cross entropy loss, we’ll be adopting the more common approach and using the binary cross entropy, or logistic loss. In this setting, we have our hypothesis function, $h_\theta(x) = w^Tx + b$.

Is the hypothesis in logistic regression simply a linear function?

I feel like this violates the definition of hypothesis, which is a function that maps to the space containing the labels. Since the labels are $\{\pm 1\}$ for binary classification, therefore it should make sense to take the hypothesis function as $h_\theta(x) = \text{sigmoid}(w^Tx + b)$ or even better, $h_\theta(x) = \text{sign}(\text{sigmoid}(w^Tx + b) - 0.5)$.

Can someone chime in on whether the choice of hypothesis is correct in the article?


See Shwartz, Ben David Understanding Machine Learning

enter image description here

$\endgroup$
  • $\begingroup$ Logistic regression has nothing to do with a hypothesis. It is a direct probability model. It is important that the output of a statistical model not be 0 or 1, as a probability model deals with tendencies. You seem to be reading a literature that is reinventing many older statistical concepts and giving them confusing terminology. $\endgroup$ – Frank Harrell Sep 12 at 22:49
  • 1
    $\begingroup$ @FrankHarrell Your diagnosis is correct -- some researchers unfamiliar with statistics did waste a lot of time a few decades ago rediscovering statistical concepts. For theirs sins, we are punished with their confusing terminology, and in some circles the confusing terms are still in common usage. "Hypothesis function" is perhaps the most common and egregious. $\endgroup$ – Sycorax Sep 13 at 0:01
  • $\begingroup$ @Sycorax Thank you. In ML, the hypothesis function is the function you model is trying to learn. After learning it, the hypothesis function should make the correct prediction for every example for which it was used during training, and should make correct predictions for similar examples. How is this terminology different from hypothesis function in statistics? $\endgroup$ – Shamisen Expert Sep 13 at 0:05
  • $\begingroup$ @ShamisenExpert Yes, I'm familiar with a "hypothesis function." The term "hypothesis" in statistics refers to hypothesis-testing, which I think is what Dr. Harrell means when he writes "Logistic regression has nothing to do with a hypothesis," but perhaps he'll clarify if my interpretation is incorrect. $\endgroup$ – Sycorax Sep 13 at 0:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.