It's always a good idea to graph problems like this first. To understand
what to graph exactly, we proceed as follows: First, we can prepare
the graph by initially "framing-out" the area support (or area
where there is positive probability under the joint probability density
function or PDF). In this case both $X$ and $Y$ are independent
and uniformly distributed on the interval $[0,1]$, which implies
that the joint PDF is uniform with PDF given by:
\begin{eqnarray*}
f_{X,Y}(x,y) & = & \begin{cases}
1, & 0<x<1;\,0<y<1\\
0, & \text{otherwise}
\end{cases}
\end{eqnarray*}
since the joint PDF of two independent random variables is simply
the product of the PDFs of each of the random variables. Now, we know
that the area of positive probability is the area bounded by the unit
square so this can be drawn on our graph (the light purple lines shown
in the graph shown below).
Now you are being asked to find the probability
that $X$ and $Y$ satisfy $X+Y<1$ and $XY<\frac{1}{10}$. After
re-arranging terms, in symbols, this becomes
\begin{eqnarray*}
P\left(X+Y<1\,\text{and}\,XY<\frac{1}{10}\right) & = & P\left(Y<1-X\,\cap\,Y<\frac{1}{10X}\right)
\end{eqnarray*}
Geometrically, this amounts to finding the density under the area
bounded by the inequalities $y<1-x$ and $y<\frac{1}{10x}$. We can
draw on our graph the line of $y=1-x$ and shade in pink the area
where $y$ is less than this line. This is shown as the areas marked
as III and IV on the figure below. Similarly, we can draw the equation
$y=\frac{1}{10x}$ in the area of support and shade in light blue
the area under this equation (areas I, III, and V). Now, note that
we are only interested in the areas that are under both $y<1-X$
and $y<\frac{1}{10x}$, within the unit square which is the area where
the pink and blue shading overlaps in the figure, or the area by III
and inside the unit square. The probability associated with this area
can be determined in two ways:
- We can perform single-variable integration in three pieces (or through
some other single-variable integral with subtraction) and sum the
resulting areas; or
- We can perform double-integration of area IV and then subtract this
from the area of the triangle formed by $y<1-x$ and the unit square
(areas III and IV).
I'll show the calculation of both Methods $1$ and $2$.
Method 1
Because the joint PDF is uniform over the unit square, then the probability
we are seeking is simply the sum of the area of interest since the
density over the entire area of support is constant. Note this method
only works in cases where the joint density is uniform over the area
of positive probability. To use this method, we have to find a suitably
smooth function over which we can integrate. Unfortunately, this doesn't
exist here. However, we can partition the area we are interested in
integrating at point of discontinuity or where the function under
which we want to integrate changes so that the function is piecemeal
smooth. This means we can break the graph into three places by
dropping a vertical line at those points where the graph of $y=1-x$
and $y=\frac{1}{10x}$ intersect (the red and blue dots on Figure
1). The second graph, Figure 2, shows these points with the corresponding
vertical lines dropped to the $x$-axis that divide up III into the
three areas marked by A, B, and C.
So Areas A and C can be obtained
by integrating under $y=1-x$ from $x=0$ to the first intersecting
points and then from the second intersecting point to $x=1$, respectively.
Area B can be obtained by integrating under the equation $y=\frac{1}{10x}$
between the two points where it intersects with the line given by
$y=1=x$ (the red and blue points).
Before we can proceed with integration we must obtain the bounds of
integration for A, B, and C, so we need to find the $x$-values where
the equations $y=1-x$ and $y=\frac{1}{10x}$ intersect. This can
be found by setting the right-hand-side (RHS) of each equation equal
to one another and then solving for $x$ through the use of the quadratic
formula:
\begin{eqnarray*}
1-x & = & \frac{1}{10x}\\
& \implies & x(1-x)=\frac{1}{10}\\
& \implies & x^{2}-x+\frac{1}{10}=0\\
& \implies & x=\frac{1-\sqrt{3/5}}{2}=\frac{\sqrt{5}-\sqrt{3}}{2\sqrt{5}}\,\text{or}\,x=\frac{1+\sqrt{3/5}}{2}=\frac{\sqrt{5}+\sqrt{3}}{2\sqrt{5}}
\end{eqnarray*}
By substitution, we find that the red point in the graphic is given
by $\left(\frac{1-\sqrt{3/5}}{2},\,\frac{1+\sqrt{3/5}}{2}\right)$
and the blue point by $\left(\frac{1+\sqrt{3/5}}{2},\,\frac{1-\sqrt{3/5}}{2}\right)$.
Now, we can finally integrate areas A, B, and C with the boundaries
of integration appropriately set:
Area A:
\begin{eqnarray*}
\int_{0}^{\frac{1-\sqrt{3/5}}{2}}1-xdx & = & \left.x-\frac{x^{2}}{2}\right|_{0}^{\frac{1-\sqrt{3/5}}{2}}\\
& = & \frac{\sqrt{5}-\sqrt{3}}{2\sqrt{5}}-\frac{5-2\sqrt{5}\sqrt{3}+3}{40}\\
& \approx & 0.1063508327
\end{eqnarray*}
Area B:
\begin{eqnarray*}
\frac{1}{10}\int_{\frac{1-\sqrt{3/5}}{2}}^{\frac{1+\sqrt{3/5}}{2}}x^{-1}dx & = & \left(\frac{1}{10}\right)\left.x^{-1}\right|_{\frac{1-\sqrt{3/5}}{2}}^{\frac{1+\sqrt{3/5}}{2}}\\
& = & \left(\frac{1}{10}\right)\log\left(\frac{1+\sqrt{3/5}}{2}\right)-\left(\frac{1}{10}\right)\log\left(\frac{1-\sqrt{3/5}}{2}\right)\\
& = & \left(\frac{1}{10}\right)\log\left[\left(\frac{1+\sqrt{3/5}}{2}\right)\left(\frac{2}{1-\sqrt{3/5}}\right)\right]\\
& = & \left(\frac{1}{10}\right)\log\left(\frac{\sqrt{5}+\sqrt{3}}{\sqrt{5}-\sqrt{3}}\right)\\
& \approx & 0.2063437069
\end{eqnarray*}
Area C:
\begin{eqnarray*}
\int_{\frac{1+\sqrt{3/5}}{2}}^{1}1-xdx & = & \frac{1}{2}-\frac{\sqrt{5}+\sqrt{3}}{2\sqrt{5}}+\frac{5+2\sqrt{5}\sqrt{3}+3}{40}\\
& = & \frac{5+2\sqrt{5}\sqrt{3}+3}{40}-\frac{\sqrt{3}}{2\sqrt{5}}\\
& \approx & 0.0063508327
\end{eqnarray*}
So the total area in III, the probability of interest, and the solution
to the problem is given by $\displaystyle{A+B+C=\frac{1}{2}\left(1-\sqrt{\frac{3}{5}}\right)+\frac{1}{10}\log\left(\frac{\sqrt{5}+\sqrt{3}}{\sqrt{5}-\sqrt{3}}\right)\approx0.3190453723}$
Method 2
When you can use double integrals, things become much easier. To use
a double integral, you essentially need to find bounds in the $x$-direction
and $y$-direction for the region you are interested in integrating
the probability density function over (region III). In the $x$-direction,
the bounds go from 0 to 1. The lower bound in the $y$-direction is
the $x$- axis. However, again, we see the upper bound in the $y$-direction
is not smooth because of the sudden changes in the bounds of integration
at the break points previously described. But, if you look closely,
you will notice that area IV has a smooth function on both the upper-
and lower-bounds in the $y$-direction and there are single points
in the $x$-direction. So we can find the area of III by finding the
area of the pink triangle (area III and IV) by using the simple geometry
formula $\frac{1}{2}\times \text{base}\times \text{height}$ (you could also use
integration here, but why do more work?) and subtract out the area
of IV found through double-integration. The resulting value will be
the probability we area after. So let's start this method by finding
the area in IV. We know already that this area is bounded on the top
by $y=1-x$ and on the bottom by $y=\frac{1}{10x}$. For the bounds
of $x$ we simply examine the graph and see area IV goes no further
to the left than the red point given by $\left(x=\frac{1-\sqrt{3/5}}{2},\,y=\frac{1+\sqrt{3/5}}{2}\right)$
and no further to the right than the blue point given by $\left(x=\frac{1+\sqrt{3/5}}{2},\,y=\frac{1-\sqrt{3/5}}{2}\right)$.
The $x$-values at these points will serve as the upper and lower
bounds in the direction of $x$. So now, we proceed to computing the
integral, the area of IV, or the probability of interest under the
density function $f_{X},_{Y}(x,y)=1$ bounded by the region given
by IV:
\begin{eqnarray*}
\text{Area IV} & = & \int_{\frac{1-\sqrt{3/5}}{2}}^{\frac{1+\sqrt{3/5}}{2}}\int_{\frac{x^{-1}}{10}}^{1-x}1dydx\\
& = & \int_{\frac{1-\sqrt{3/5}}{2}}^{\frac{1+\sqrt{3/5}}{2}}1-x-\frac{x^{-1}}{10}dx\\
& = & \left.x-\frac{x^{2}}{2}-\frac{1}{10}\log(x)\right|_{\frac{1-\sqrt{3/5}}{2}}^{\frac{1+\sqrt{3/5}}{2}}\\
& = & \left[\frac{1+\sqrt{3/5}}{2}-\left(\frac{1}{2}\right)\left(\frac{1+\sqrt{3/5}}{2}\right)^{2}-\frac{1}{10}\log\left(\frac{1+\sqrt{3/5}}{2}\right)\right]-\left[\frac{1-\sqrt{3/5}}{2}-\left(\frac{1}{2}\right)\left(\frac{1-\sqrt{3/5}}{2}\right)^{2}-\frac{1}{10}\log\left(\frac{1-\sqrt{3/5}}{2}\right)\right]\\
& = & \frac{\sqrt{3}}{\sqrt{5}}-\frac{\sqrt{3}\sqrt{5}}{10}+\frac{1}{10}\log\left(\frac{\sqrt{5}-\sqrt{3}}{\sqrt{5}+\sqrt{3}}\right)\\
& = & \frac{\sqrt{3}}{2\sqrt{5}}+\frac{1}{10}\log\left(\frac{\sqrt{5}-\sqrt{3}}{\sqrt{5}+\sqrt{3}}\right)
\end{eqnarray*}
And finally, to calculate the probability of interest, we simply need
to subtract the probability calculated immediately above (Area IV)
from the area of the triangle with vertices at $(0,0)$, $(1,0)$, and $(0,1)$
(Area III and IV):
\begin{eqnarray*}
\text{Area III} & = & \text{Area III}+\text{Area IV}-\text{Area IV}\\
& = & \text{Triangle with vertices at}\,(0,0)\,(0,1)\,(1,0)-\text{Area IV}\\
& = & \frac{1}{2}\left(\text{base}\right)\left(\text{height}\right)-\text{Area IV}\\
& = & \frac{1}{2}(1)(1)-\left[\frac{\sqrt{3}}{2\sqrt{5}}+\frac{1}{10}\log\left(\frac{\sqrt{5}-\sqrt{3}}{\sqrt{5}+\sqrt{3}}\right)\right]\\
& = & \frac{\sqrt{5}-\sqrt{3}}{2\sqrt{5}}-\frac{1}{10}\log\left(\frac{\sqrt{5}-\sqrt{3}}{\sqrt{5}+\sqrt{3}}\right)\\
& \approx & 0.3190453723
\end{eqnarray*}
and this is the same answer as the one we obtained using Method 1.$\blacksquare$