What is the optimal bandwidth in this problem Suppose that I have some estimator $\hat f$ for the parameter $f$. I obtain the following convergence rate $$E\|\hat f - f\|^2 = O\left(\frac{1}{nh} + \frac{1}{nh^2} + \frac{1}{\sqrt{n}h} + h^{2}\right)$$ where $h$ is a bandwidth parameter. How can I discuss what is the optimal value of the bandwidth parameter? Is it interpretable in terms how fast should $h\to 0$ as $n\to\infty$?
In the density estimation I have $E\|\hat f - f\|^2 = O\left(\frac{1}{nh} + h^4\right)$, so it is easy to discuss the optimal bandwidth. Taking first-order conditions, we can see that, we need $h\sim n^{-1/5}$ and so $E\|\hat f - f\|^2\sim n^{-4/5}$.
 A: It should be evident from the expressions that the optimal solution will be of the form $h = h(n) = n^\alpha$, because then all four terms become powers of $n$.  (Details are given below.)  Written in this form they become
$$ O\left(\frac{1}{nh} + \frac{1}{nh^2} + \frac{1}{\sqrt{n}h} + h^{2}\right) =  O\left(n^{-1-\alpha} + n^{-1-2\alpha} + n^{-1/2-\alpha} + n^{2\alpha}\right).$$
The asymptotics will be the best possible when the largest of these exponents is as small as possible, because it will asymptotically dominate the others. We seek therefore to minimize
$$\max\{-1-\alpha, -1-2\alpha, -1/2-\alpha, 2\alpha\}$$
An easy way to solve this minimax problem is to graph these four (linear) functions of $\alpha$.   Clearly we will need $\alpha\lt 0$ (so that $h$ decreases as $n$ grows), allowing us to limit the plot to small negative $\alpha$: 

Blue, red, green, and gold correspond to the four exponents in the order listed.
By shading the points beneath the four curves (all of which are dominated by at least one of the curves), we reveal the optimum as the lowest unshaded point in the plane.  It occurs where $-1/2-\alpha = 2\alpha$, whence $\alpha=-1/6$ is the unique solution.  The four exponents are
$$\{-1-\alpha, -1-2\alpha, -1/2-\alpha, 2\alpha\} = \{-5/6, -2/3, -1/3, -1/3\}$$
(The last two constraints are the active ones; the first two constraints are not relevant.)  Their maximum is $-1/3$.  (The point $(-1/6, -1/3)$ is shown as a black dot in the figure.)  The optimal convergence rate is 
$$O(n^{-1/3}).$$

(Edit)
Why should the optimal solution have the form $n^\alpha$?  In general all we know is that it should be of the form $n^\alpha f(n)$ where, as $n\to\infty$, either $f(n)/n^\beta \to 0$ or $f(n)/n^\beta \to \infty$ for any $\beta\ne \alpha$.  An improvement is possible if $f(n) = o(n)$; that is, if $f(n)\to 0$ as $n\to \infty$.
The foregoing analysis shows that the two terms controlling the optimum are the last two, so let's consider how $f$ might affect them:
$$O(\ldots + n^{-1/2}/h(n) + h(n)^2) = O(n^{-1/2}n^{-\alpha}/f(n) + n^{2\alpha}f(n)^2) \\ = O\left(n^{-1/3}\left(\frac{1}{f(n)} + f(n)^2\right)\right).$$
If $f(n)=o(n)$, the first term becomes arbitrarily large, demonstrating it is not possible to improve on the solution $h(n)=n^{-1/6}$.
