I'll provide the more intuitive approach by graphing the problem as suggested by the other users and using properties of uniform random variables.
Before I get started, one thing I have noticed that might be causing you a lot of confusion is that you have $P(A^2<B)$ in your title, but in the body of your post you show that you trying to find the compliment of this, namely $P(A^2>B)$ (at least that's what you wrote the third time you wrote the probability inequality right after you have "Approach 2" above). I'm not sure which one you are looking for, so I'll assume that you are trying to obtain what you have written first in your title: $P(A^2<B)$ (note that @Ben assumed the opposite, so the answer will be different: 1 minus the answer he provided, in fact).
To get started, you should always draw a graph of the problem. As @whuber appropriately stated in the comments, "the understanding and computation of multidimensional integrals are greatly enhanced by drawing a picture." In this picture, I'm going to chose two different values for $h$: one less than $h=1$ and one greater than $h=1$. In this case, I have chosen $h=0.6$ and and $h=2$. Then, I'll add a graph of $a^2<b$ and shade the region of integration separately for the $h=0.6$ and $h=2$ cases. The graphs appear below with the boundary for $h<1$ (in our example, $h=0.6$) in green and with the boundary for $h>1$ (in our example, $h=2$) in light blue.
What you should immediately notice is the $a^2<b$, shown in red, cuts through the side of the right-most, green boundary when $h < 1$ and it cuts through the light blue, top boundary when $h>1$. The $b$-axis forms the other boundary of integration. The area of integration is shaded in each scenario. Now, all we have to do is read off our graph to form the integrals for each case:
For $h\le1$:
The bounds of integration can be calculated by breaking the shaded part into two pieces where $a^2=b$ and $b=h$ intersect. But I think it's easier to calculate $I$ where $I$ = the area under the red curve and to the left of $h$ (the vertical green line) and then calculate $h^2 - I$. Since $h^2$ is the total area, $h^2 - I$ is the shaded area of interest. So:
\begin{eqnarray*}
I & = & \int_{0}^{h}\int_{0}^{a^{2}}dbda\\
& = & \int_{0}^{h}a^{2}da\\
& = & \frac{h^{3}}{3}
\end{eqnarray*}
So the shaded region for $h\le1$ is given by:
\begin{eqnarray*}
\text{Shaded Region} & =h^{2}- & \frac{h^{3}}{3}
\end{eqnarray*}
Now, since both $A$ and $B$ are uniform random variables, we can easily calculate $P(A^2<B)$ by taking the area we found above and dividing by the total area of support, $h^2$ (note this does not generalize to cases where $A$ and $B$ are not uniform - this is a special property of sort of uniform random variables). So:
\begin{eqnarray*}
P(A^{2}<B) & = & \frac{h^{2}-\frac{h^{3}}{3}}{h^{2}}\\
& = & 1-\frac{h}{3}
\end{eqnarray*}
So we are done with the case for $h\le1$. Now we move on to the case where $h>1$ (the graph on the right).
For $h \gt 1$:
The area of integration for $h\gt1$ is easier than for $h\le1$ if you wisely chose which direction to integrate in first. If you choose to integrate in the direction of $a$ first, integration is straight forward and the area shaded in the figure on the right is given by:
\begin{eqnarray*}
\int_{0}^{h}\int_{0}^{b^{\frac{1}{2}}}dadb & = & \int_{0}^{h}b^{\frac{1}{2}}db\\
& = & \frac{2}{3}b^{\frac{3}{2}}\Biggr\rvert_{b=0}^{b=h}\\
& = & \frac{2h^{\frac{3}{2}}}{3}
\end{eqnarray*}
Again, since both $A$ and $B$ are uniform random variables, we can easily calculate $P(A^2<B)$ for $h>1$ by taking the area we found above and dividing by the total area of support, $h^2$:
\begin{eqnarray*}
\frac{2h^{\frac{3}{2}}}{3h^{2}} & = & \frac{2h^{\frac{3}{2}}h^{-2}}{3}=\\
& = & \frac{2h^{\frac{3}{2}}h^{-\frac{4}{2}}}{3}\\
& = & \frac{2h^{-\frac{1}{2}}}{3}\\
& = & \frac{2}{3h^{\frac{1}{2}}}\\
& = & \frac{2}{3\sqrt{h}}
\end{eqnarray*}
Combining the answers above, we get:
\begin{eqnarray*}
P(A^{2}<B) & = & \begin{cases}
1-\frac{h}{3} & \text{for $h\le1$}\\
\\
\frac{2}{3\sqrt{h}} & \text{for $h>1$}
\end{cases}
\end{eqnarray*}
That's all there is to it!