Why must a null hypothesis contain equality? Using the inference of mean as an example, the null and alternative hypothesis could be
$$H_0: \mu \le 0 \Leftrightarrow H_1: \mu > 0$$
It is often argued that this makes the calculation of the $p$-value (or critical region) easier, but from my understanding the $p$-value is $P(\bar{X} > 0|H_0) \stackrel{\color{red}{?}}{=} \sup_{\theta \in H_0} P(\bar{X} > 0 | \theta )$. Since I'm taking its supremum, the $p$-value doesn't change regardless of whether the equality sign is included. The same result goes for the critical region. However, I'm not sure if my understanding is correct, and my doubt is marked with a red question mark above.
Anyway, I wonder if it's correct (I know that would be unconventional, though) to write
$$H_0: \mu < 0 \Leftrightarrow H_1: \mu \ge 0$$
If that's OK, what about its $p$-value and critical region? If that's not OK, I would be really surprised, because in that way we are changing the question ("Is $\mu$ less than or equal to zero?") to something different ("Is $\mu$ less than zero?")
 A: The principle of the hypothesis test requires the computation of the rejection probability assuming H0 (type I error probability). In fact, if the null hypothesis is not just one point, we are fine with the supremum, so that we can state that the type I error probability of the test is smaller or equal to that value. So far this would be in line with what you're writing and wouldn't require the equality case to be included.
However, if in fact the rejection probability for the "equal" is equal to the supremum over the rest of the null hypothesis (as is the case for most tests although one could imagine tests for which this doesn't hold), the test will not distinguish the equality case from the H0, i.e., you will not have a higher rejection probability for "equal" than for the H0 as a whole, and therefore it makes sense to include "equal" in the H0.
In the words of E.S. Pearson (if I remember correctly), tests for which parameter values in the H1 have a lower rejection probability than cases in the H0 are "worse than useless", and if they have the same rejection probability, they are just useless. 
Why would you want to reject a hull hypothesis in favour of an alternative that has a rejection probability that isn't higher? The idea of tests is that you collect evidence in favour of the alternative from observing a rejection event that has a probability under the alternative that is higher than under the H0.
PS (added later): Technically, according to the general definition of hypothesis tests, you do not always have to include $\mu=0$ in the null hypothesis (note also that not all tests are Neyman-Pearson tests). I'm not giving you a reason why this isn't permitted, I'm giving you a reason why in most cases it doesn't make sense. As written above, I can even imagine to construct a problem with a weird model and parametrisation according to which the rejection probability at $\mu=0$ is in fact higher than the supremum over $\mu<0$, in which case I wouldn't object at all against testing $H_0: \mu<0$ against $H_1: \mu\ge 0$. However pretty much all the standard tests that you find in the literature are not of this kind. In all the standard cases in fact the test level is computed on the hypothesis border, i.e. (using the parametrisation referred to in the question), $\mu=0$, and then $\mu<0$ is attached based on the fact that the supremum over those parameter values is not bigger. So it would be illogical to then attach  $\mu=0$, the basis of the level computation under H0, to the H1.   
PPS, prompted by your comment: In my (hopefully not too unconventional) terminology the "rejection probability" is the probability, given $\mu$, to reject the H0. This depends on $\mu$. Particularly, if $\mu$ is in the H0, it is a type I error probability, whereas if $\mu$ is in the H1, it is the power. In a good test, you want the type I error probability to be low and the power to be high. A test in which the power is not higher than the type I error probability (or be it the supremum of these) doesn't do what it's supposed to do, it doesn't indicate the H1 in case of rejection. Or let's say it doesn't indicate the $\mu=0$ part of the H1 in case you chose to include $\mu=0$ in the H1. It's nonsense to say "this level $\alpha$-test indicates evidence in favour of $\mu\ge 0$" if the power in case of $\mu=0$ is no better than that $\alpha$.  
A: In traditional hypothesis testing the null hypothesis always contains
an $=$-sign, whether it is as $=, \le,$ or $\ge.$
The mull hypothesis determines the null distribution of the test statistic. Hence also the critical value used in testing at a particular level or, in computer programs, the P-value.
Example: In a simple binomial test whether a coin is fair, when suspected
of bias towards Heads, might be $H_0: p = 1/2$ vs $H_0: p > 1/2.$
Suppose data consist of 100 coin tosses: We can reject $H_0$ at the 4.43%
level, if the number $X$ of Heads matches or exceeds the 'critical value' $c= 59.$ (Because of the discreteness of the binomial distribution a nonrandomized test at exactly the 5% level is not available.) 
In R, $P(X \ge 59\,|\,p=.5) =
1 - P(X \le 58\,|\,p = .5) = 0.0443.$ 
1 - pbinom(58, 100, .5)
[1] 0.04431304


Intuitively, it might seem that getting 55 Heads in 100 should arouse suspicion
that the coin is biased for Heads. "Suspicion" maybe Yes, but this would not be
a statistically significant result at the stated level of significance.
By random variation a truly fair coin can show 55 or more Heads in 100 tosses with probability about 0.184.
1 - pbinom(54, 100, .5)
[1] 0.1841008

Also, if we observe $X = 62$ Heads then the P-value is $P(X \ge 62\,|\,p=.5) = 0.01,$ leading to rejection of $H_0$ because the P-value $0.0105 < 0.0443.$
The P-value is defined as the probability, under $H_0,$ of an outcome more
extreme (in the direction of the alternative) than what was observed.
1 - pbinom(61, 100, .5)
[1] 0.01048937

If the hypotheses were formulated as $H_0: p \le 1/2$ vs. $H_0: p > 1/2,$
then the $=$-sign in $H_0$ would still govern these probability computations.
