I'm learning about logistic regression from the Andrew Ng Machine Learning course. In the course it is stated that the hypothesis is the probability of the output $y=1$ for a given input $x$. That is, $$h_\theta(x)=P(y=1|x;\theta).$$
It is assumed that $h_\theta(x) = \frac{1}{1+e^{-\theta^T x}}$, i.e. the sigmoid function.
What I am confused about is how we know this? Is this a definition?
There are many "sigmoid" functions, but the one you quote, the logistic function, is the most common in machine learning and statistics. I can think of two reasons for it:
The first reason is likely to be given more often by statisticians and the second by machine learners.
1. Statistical approach
Assume a random variable $Y$ which can take two discrete values, say $A$ and $B$---these can be regarded as "class labels". Define a variable $y$ which is one when $Y=B$ and zero otherwise (i.e. $Y=A \Leftrightarrow y=0$). Assume also that the probability of a point with a real (possibly vector) value $x$ to belong to the class $B$ depends somehow on that value, i.e. $P(y) = p(x)$, with $p(x)$ being a function of $x$, $p : \mathbb{R} \rightarrow [0, 1]$.
With some mathematical acrobatics, $P(y)$ can be rewritten to fit into the general notation for the exponential family of probability distributions:
$$ P(y) = h(y) \exp(\eta(x) \cdot T(y) - A(\eta(x))) $$
Depending on the choice of $h(y)$, $\eta(x)$, $T(y)$ and $A(\eta(x))$ you can construct many different distributions: Gaussian, Poisson, Gamma etc. For our distribution ("Bernoulli"), we need to set:
$$ \begin{align} h(y) &= 1\\ \eta(x) &= \ln \frac{p(x)}{1-p(x)} \\ T(y) &= y \\ A(\eta) &= \ln \left( 1 + \exp(\eta(x)) \right) \end{align} $$
For the moment, you can safely ignore everything except the second line, defining $\eta(x)$. This relationship is called log-odds. Now, if we express $p(x)$ in terms of $\eta(x)$ we get:
$$ p(x) = \frac{1}{1 + e^{-\eta(x)}} $$
Generally, $\eta(x)$ can have any form, but it is mathematically appealing to assume it to be a linear function, or, in your notation, $\eta(x) = \theta^T x$. Hence the "generalised linear model".
2. Machine learning approach
Assume two probability distributions ("classes"), $A$ and $B$. Regardless of their probability distributions, it follows from the Bayesian formula:
$$ P(B | x) = \frac{P(x | B) P(B)}{P(x)} = \frac{P(x | B) P(B)}{P(x | A) P(A) + P(x | B) P(B)} = \frac{1}{1 + \frac{P(x | A)P(A)} {P(x | B)P(B)}} $$
If $x$ is continuous, so that the classes can be described by their respective probability density functions (PDFs), $f_A(x)$ and $f_B(x)$, the fraction $P(x | A) / P(x | B)$ can be expressed as:
$$ \frac{P(x | A)} {P(x | B)} = \lim_{\Delta x \rightarrow 0} \frac{f_A(x) \Delta x}{f_B(x) \Delta x} = \frac{f_A(x)}{f_B(x)} $$
Now, let us assume that the two classes are normally distributed, with equal variances:
$$ f_A(x) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left( -\frac{(x - \mu_A)^2} {2 \sigma^2} \right), ~ ~ ~ ~ ~ ~ ~ ~ ~ f_B(x) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left( -\frac{(x - \mu_B)^2} {2 \sigma^2} \right) $$
Then the fraction $f_A(x) / f_B(x)$ can be written as:
$$ \frac{f_A(x)}{f_B(x)} = \exp \left( - \frac{(x - \mu_A)^2} {2 \sigma^2} + \frac{(x - \mu_B)^2} {2 \sigma^2} \right) = \exp \left( \frac{\mu_B^2 - \mu_A^2} {2 \sigma^2} + \frac{\mu_A - \mu_B} {\sigma^2} x \right) $$
and the whole term
$$ \frac{f_A(x)P(A)}{f_B(x)P(B)} = \exp \left( \ln \frac{P(A)}{P(B)} + \frac{\mu_B^2 - \mu_A^2} {2 \sigma^2} + \frac{\mu_A - \mu_B} {\sigma^2} x \right) $$
Denoting
$$ \theta_0 = \frac{\mu_A^2 - \mu_B^2} {2 \sigma^2} - \ln \frac{P(A)}{P(B)} ~ ~ ~ ~ ~ ~ ~ ~ \text{and} ~ ~ ~ ~ ~ ~ ~ ~ \theta_1 = \frac{\mu_B - \mu_A} {\sigma^2} $$
leads to the form commonly used in logistic regression:
$$ P(B | x) = \frac{1}{1 + e^ {-\left(\theta_0 + \theta_1 x \right)} } $$
or, in vector notation and assuming $x$ with the zeroth element fixed to 1:
$$ P(B | x) = \frac{1}{1 + e^ {-\theta^T x} } $$
Whichever approach you take, you are making assumptions about the data. In the statistical approach, you assume that the log-odds depend linearly on $x$. In the machine learning approach, you assume normal distributions of the classes (the log-odds assumption follows from the normality assumption, but not the other way round). In practice, normally distributed data are quite common, so this assumption is likely, but not guaranteed to be true. If not, your modeled probabilities will deviate from the true probabilities.
Because probabilities are bounded between 0 and 1, we need some function which can take the unbounded input $\theta^T x$ and map it to the unit interval. Mathematically, we need
$$f : \mathbb{R} \to [0,1]$$
The sigmoid function accomplishes this, but it is not the only one. If we had used the CDF of the normal, we would be doing something called Probit regression (closely related to logistic regression).
In short, we have constructed logistic regression (i.e. defined it) in a way that $P(y=1) = h_\theta (x)$.
