If by open/closed you mean $[a,b]$ vs $(a,b)$, then it's in a continuous domain it doesn't make a difference. Consider a continuous pdf defined in the domain $a$ to $b$. The integral over $[a,b]$ will be equal to the integral over $(a,b)$ because the integral over a single point is zero, so excluding any countable set of points from the integrand won't change its value at all.
Now, on to some philosophy: generally, our null hypothesis is either an assertion that some population parameter is the same across treatments, or that the parameters are within some defined tolerance of each other. Since we are fixing this tolerance, it makes sense to define it with a closed set, where the set is closed up to the maximum tolerance, e.g. $H_0: \theta \le \theta_0$ where $\theta_0$ defines the maximum allowable tolerance. Since we are parameterizing our hypothesis with respect to the maximum allowable tolerance, it makes sense to use closed notation here. But, as described above, this hypothesis is functionally equivalent to $H_0: \theta \lt \theta_0$, but the interpretation is a little weirder now: $\theta_0$ now denotes the minimum rejection value of the parameter, so the allowable tolerance is infinitesimally close to but not equal to $\theta_0$. I think you'll agree that it generally makes more sense for the purposes of interpretation to define the null hypothesis with respect to the permissible range of parameter values.
If you meant something different by closed vs. open (maybe you meant it in some technical topological sense that I missed), please elaborate.