Think about this geometrically: we've got $(X,Y)$ uniformly distributed over the right triangle with vertices at $(0,0)$, $(2,0)$, and $(0,6)$. 

If we imagine sampling over and over from this region uniformly, we can picture $E(X|Y=5)$ as the average $x$ coordinate of the points that end up on the horizontal line $y=5$. All of these points will necessarily have $0 < x < 1/3$, since if $x \geq 1/3$ then $y = 5$ can't happen, and since the points are uniform over this line, this image suggests $E(X|Y=5) \stackrel ?= 1/6$ (this is the midpoint of the range of $x$ points that can end up on this line).

Checking this with calculus, the issue with what you did is the limits of integration for $x$ also depend on $y$. If $y = 5$ is observed then the biggest that $x$ can be is $1/3$, so really
$$
E(X|Y=5) = \int_0^{1/3}x \frac{3}{6-y}\,\text dx = 3 \cdot \frac 12 x^2\bigg|_{x=0}^{x=1/3} = \frac 16.
$$