Find $P(A^2 < B)$ where $A$ and $B$ are independent and uniformly distributed $\mathrm{Unif}(0,h)$, $h > 0$ I solved it two ways and in both the cases the answer is different and different from the actual answer.
Approach 1:
Since, $A$ and $B$ are independent, we can find the joint distribution of $AB$ which can be written as
$f_{AB} = \frac{1}{h^2}$
To find the required probability we can integrate the joint probability over the correct range for $P(B > A^2)$:
$$\int_{a = 0}^{h}\int_{b = a^2}^{h} \frac{1}{h^2} db da = 1 - \frac{h}{3}.$$
Approach 2:
$$P(A^2 > B) = P(-\sqrt{B} < A < \sqrt{B}).$$
We know $B$ and $A$ are never negative and hence we can write it as $P(0 < A < \sqrt{B})$. We can now integrate  
$$\int_{b=0}^{h}\int_{a = 0}^{\sqrt{b}} \frac{1}{h^2} da db = \frac{2}{3\sqrt{h}}.$$
However, none of the approach seems to give the result that matches with the solution given to me:
$\frac{2h}{3}$ for $h \le 1$
$\frac{2}{3\sqrt{h}}$ for $h > 1$
I would like to know what am I missing?
 A: I'll provide the more intuitive approach by graphing the problem as suggested by the other users and using properties of uniform random variables.
Before I get started, one thing I have noticed that might be causing you a lot of confusion is that you have $P(A^2<B)$ in your title, but in the body of your post you show that you trying to find the compliment of this, namely $P(A^2>B)$ (at least that's what you wrote the third time you wrote the probability inequality right after you have "Approach 2" above). I'm not sure which one you are looking for, so I'll assume that you are trying to obtain what you have written first in your title:  $P(A^2<B)$ (note that @Ben assumed the opposite, so the answer will be different:  1 minus the answer he provided, in fact).
To get started, you should always draw a graph of the problem.  As @whuber appropriately stated in the comments, "the understanding and computation of multidimensional integrals are greatly enhanced by drawing a picture."  In this picture, I'm going to chose two different values for $h$:  one less than $h=1$ and one greater than $h=1$.  In this case, I have chosen $h=0.6$ and and $h=2$.  Then, I'll add a graph of $a^2<b$ and shade the region of integration separately for the $h=0.6$ and $h=2$ cases. The graphs appear below with the boundary for $h<1$ (in our example, $h=0.6$) in green and with the boundary for $h>1$ (in our example, $h=2$) in light blue.

What you should immediately notice is the $a^2<b$, shown in red, cuts through the side of the right-most, green boundary when $h < 1$ and it cuts through the light blue, top boundary when $h>1$.  The $b$-axis forms the other boundary of integration.  The area of integration is shaded in each scenario.  Now, all we have to do is read off our graph to form the integrals for each case:
For $h\le1$:
The bounds of integration can be calculated by breaking the shaded part into two pieces where $a^2=b$ and $b=h$ intersect.  But I think it's easier to calculate $I$ where $I$ = the area under the red curve and to the left of $h$ (the vertical green line) and then calculate $h^2 - I$.  Since $h^2$ is the total area,  $h^2 - I$ is the shaded area of interest.  So:
\begin{eqnarray*}
I & = & \int_{0}^{h}\int_{0}^{a^{2}}dbda\\
 & = & \int_{0}^{h}a^{2}da\\
 & = & \frac{h^{3}}{3}
\end{eqnarray*}
So the shaded region for $h\le1$ is given by:
\begin{eqnarray*}
\text{Shaded Region} & =h^{2}- & \frac{h^{3}}{3}
\end{eqnarray*}
Now, since both $A$ and $B$ are uniform random variables, we can easily calculate $P(A^2<B)$ by taking the area we found above and dividing by the total area of support, $h^2$ (note this does not generalize to cases where $A$ and $B$ are not uniform - this is a special property of sort of uniform random variables).  So:
\begin{eqnarray*}
P(A^{2}<B) & = & \frac{h^{2}-\frac{h^{3}}{3}}{h^{2}}\\
 & = & 1-\frac{h}{3}
\end{eqnarray*}
So we are done with the case for $h\le1$.  Now we move on to the case where $h>1$ (the graph on the right).
For $h \gt 1$:
The area of integration for $h\gt1$ is easier than for $h\le1$ if you wisely chose which direction to integrate in first.  If you choose to integrate in the direction of $a$ first, integration is straight forward and the area shaded in the figure on the right is given by:
\begin{eqnarray*}
\int_{0}^{h}\int_{0}^{b^{\frac{1}{2}}}dadb & = & \int_{0}^{h}b^{\frac{1}{2}}db\\
 & = & \frac{2}{3}b^{\frac{3}{2}}\Biggr\rvert_{b=0}^{b=h}\\
 & = & \frac{2h^{\frac{3}{2}}}{3}
\end{eqnarray*}
Again, since both $A$ and $B$ are uniform random variables, we can easily calculate $P(A^2<B)$ for $h>1$ by taking the area we found above and dividing by the total area of support, $h^2$:
\begin{eqnarray*}
\frac{2h^{\frac{3}{2}}}{3h^{2}} & = & \frac{2h^{\frac{3}{2}}h^{-2}}{3}=\\
 & = & \frac{2h^{\frac{3}{2}}h^{-\frac{4}{2}}}{3}\\
 & = & \frac{2h^{-\frac{1}{2}}}{3}\\
 & = & \frac{2}{3h^{\frac{1}{2}}}\\
 & = & \frac{2}{3\sqrt{h}}
\end{eqnarray*}
Combining the answers above, we get:
\begin{eqnarray*}
P(A^{2}<B) & = & \begin{cases}
1-\frac{h}{3} & \text{for $h\le1$}\\
\\
\frac{2}{3\sqrt{h}} & \text{for $h>1$}
\end{cases}
\end{eqnarray*}
That's all there is to it!
A: This question can be solved via the double-integral method you are using, but your question makes some errors in the integrations.  If you would like to proceed by this method, you need to be careful of the integral ranges, and you will need to distinguish the case where $0 < h \leqslant 1$ and the case where $h > 1$.  As other users have pointed out in the comments, the simplest way to deal with double-integrals when you are a beginner is to draw the domain of integration.
To add something to this analysis, I thought it might be worth mentioning an alternative method that can be used to get the solution.  For any non-negative continuous random variable $X \geqslant 0$ and any uniform random variable $U \sim \text{U}(0,1)$ (where these are independent random variables) you have:
$$\begin{equation} \begin{aligned}
\mathbb{P}(X \geqslant U) 
&= \int \limits_0^1 \mathbb{P}(X \geqslant U|U=u) f_U(u) du \\[6pt]
&= \int \limits_0^1 \mathbb{P}(X \geqslant u) du \\[6pt]
&= \int \limits_0^1 \Bigg( \int \limits_u^\infty f_X(x) dx \Bigg) du \\[6pt]
&= \int \limits_0^\infty \Bigg( \int \limits_0^{\min(1,x)} f_X(x) du \Bigg) dx \\[6pt]
&= \int \limits_0^\infty \min(1,x) f_X(x) dx = \mathbb{E}(\min(1,X)). \\[6pt]
\end{aligned} \end{equation}$$
In your problem you have $U = B/h \sim \text{U}(0,1)$ and $X = A^2/h \geqslant 0$ which means you get:
$$\mathbb{P}(A^2 \geqslant B) 
= \mathbb{P} \Big( \frac{A^2}{h} \geqslant \frac{B}{h} \Big)
= \mathbb{E} \Big( \min \Big( 1, \frac{A^2}{h} \Big) \Big).$$
From here you can apply the law of the unconscious statistician (being careful with the integral ranges) to get:
$$\begin{equation} \begin{aligned}
\mathbb{P}(A^2 \geqslant B) 
= \mathbb{E} \Big( \min \Big( 1, \frac{A^2}{h} \Big) \Big)
&= \int \limits \min \Big( 1, \frac{a^2}{h} \Big) f_A(a) da \\[6pt]
&= \int \limits_0^h \min \Big( 1, \frac{a^2}{h} \Big) \frac{1}{h} da \\[6pt]
&= \int \limits_0^{\min(h, \sqrt{h})} \frac{a^2}{h^2} da + \mathbb{I}(h > 1) \int \limits_\sqrt{h}^h \frac{1}{h} da \\[6pt]
&= \frac{\min(h, \sqrt{h})^3}{3h^2} + \frac{h - \sqrt{h}}{h} \cdot \mathbb{I}(h > 1) \\[6pt]
&= \begin{cases}
\frac{h}{3} & & \text{if } h \leqslant 1, \\[6pt]
1 - \frac{2}{3\sqrt{h}}  & & \text{if } h > 1. \\[6pt]
\end{cases} \\[6pt]
\end{aligned} \end{equation}$$
