I will give the outline of the solution. Numbers of each coloured ball in a draw follows multinomial distribution as tshauck pointed out. Let $R$ denote the number of red balls, $G$ - number of green balls, $B$ - number of blue balls and $Y$ - number of yellow balls in the draw of the size $n$. Then the probability that in random draw we have exactly $x_1$ red balls, $x_2$ green balls, $x_3$ blue balls and $x_4$ yellow ball is
where $p_r$ is probability of picking red ball, $p_g$ - green, $p_b$ - blue, $p_y$ - yellow.
Denote the number of unique balls in a draw by $U$. Then $U=f(R,G,B,Y)$. Since you have the distribution of vector $(R,G,B,Y)$ you can calculate distribution of $U$. Since we have four colours $U$ can get values $0,1,2,3,4$. So to get probability that $U=0$ you need to find all the possible combinations of $(R,G,B,Y)$ for which $U=0$ and add the probabilities of these combinations. So when $U=0$? When
All the balls appear more than once: $R>1$, $G>1$, $B>1$, $Y>1$
One colour is absent and all others appear more than once
a. $R=0$, $G>1$, $B>1$, $Y>1$
b. $R>1$, $G=0$, $B>1$, $Y>1$
c. $R>1$, $G>1$, $B=0$, $Y>1$
d. $R>1$, $G>1$, $B>1$, $Y=0$
Two colours are absent and all others appear more than once. $6$ cases
Three colours are absent, all the draw is of one colour. $4$ cases
All the four cases are mutually exclusive, so you can add the probabilities. Cases of $U=1,2,3,4$ can be treated similarly.
This of course is not an elegant solution, but it I do not see why it should not work. I suggest asking this in math.stackexchange.com.
This approach is for calculating the probability distribution of $U$. For expected value of $U$ - whuber's answer is the right one.