A generalization
There are many ways to generalize this problem.
For instance, let there be $n$ bins indexed by $j=1,2,\ldots,n,$ each capable of holding no more than $b_j$ balls, into which $k_1$ balls of color $1$, $k_2$ balls of color $2,$ ..., and $k_m$ balls of color $m$ are to be placed in such a way that for each color $i,$ bin $j$ must contain no more than $u_{ij}$ balls of color $i.$ Let any such placement be called a "configuration:" we wish to count distinct configurations.
The example of the question is specified by vectors $\mathbf{k}=(2,2,2)$ (color counts), $\mathbf{b}=(2,2,2,2,2)$ (bin capacities), and an $m\times n=3\times 5$ matrix $(u_{ij})=(1)$ (per-color bin limits).
Limiting the generalization to achieve a tractable problem
It looks difficult, if not impossible, to produce a program, much less a formula, for this level of generality. In some special cases, however, simple formulas are possible. One of these still generalizes the example of the question. We suppose there are equal numbers $k$ of each color and each of $n$ bins must contain fewer than $m$ balls, all of different colors. Thus, $\mathbf{k}=(k,k,\ldots, k)$ is an $m$-vector, $\mathbf{b}=(m-1,m-1,\ldots,m-1)$ is an $n$-vector, and it remains the case that $(u_{ij})=1.$
Solution of the limited generalization
To count the configurations in this special case, note that without any restrictions on bin contents the disposition of $k$ monochromatic balls with no more than one per bin corresponds to the selection of $k$ bins in which to place them, of which there are $\binom{n}{k}$ possibilities. This yields $\binom{n}{k}^m$ possible configurations for all $km$ balls.
Some of these configurations are "bad" in the sense that they violate the bin capacity restriction: one or bins will contain all $m$ colors. Let's call these the "loaded" bins. We may count the bad configurations by focusing on the loaded bins.
There are $n$ possibilities for the loaded bins. That bin contains one ball of each color, leaving $k-1$ balls of each color to be placed in the remaining $n-1$ bins. There are $\binom{n-1}{k-1}$ ways to do that for each of the $m$ colors. Thus, by summing over all $n$ possible loaded bins, we find there are at most $n\binom{n-1}{k-1}^m$ bad configurations.
That sum counts any bad configurations with two or more loaded bins as many times as it has loaded bins: that's why it potentially over-counts the bad configurations. We need to adjust it. Consider the case of two loaded bins. Arguing as before, we see (a) there are $\binom{n}{2}$ distinct possible subsets of two loaded bins and (b) in such cases there remain $k-2$ balls of each color to be placed into the remaining $n-2$ bins. Thus, we should subtract $\binom{n}{2}\binom{n-2}{k-2}^m$ from the previous overestimate.
This, however, subtracts too much, because we have not accounted for ay bad configurations with three or more loaded bins. The continuation of the argument should be clear: it is an application of the Principle of Inclusion-Exclusion, or "PIE." The alternating addition and subtraction continues until there are no balls left to account for, giving the formula
$$f(n,m,k) = \sum_{i=0}^k (-1)^i \binom{n}{i}\binom{n-i}{k-i}^m.$$
Examples
(1) The problem in the question.
In the case of $n=5$ bins to be filled with balls of $m=3$ colors ($k=2$ of each color available) with at most $m-1$ balls in each bin, we find
$$f(5,3,2) = \binom{5}{0}\binom{5}{2}^3 - \binom{5}{1}\binom{4}{1}^3 + \binom{5}{2}\binom{3}{0}^3 = 10^3 - 5(4^3) + 10(1^3) = 690.$$
If we vary the numbers of bins, still keeping to two balls each of three colors, the sequence $f(3,3,2), f(4,3,2), \ldots, f(n,3,2)$ begins
$$6, 114, 690, 2640, 7770, 19236, 42084, 83880, 155430, 271590, \ldots$$
This sequence is not in the OEIS, indicating it has not specifically been studied before (although likely it is related to some well-known sequence).
(2) A binomial identity
When the total number of balls, $km,$ exceeds $n(m-1),$ the Pigeonhole Principle implies at least one bin must contain at least $m$ balls. Thus, all configurations are bad and the answer is $0.$ In such cases
$$0 = f(n,m,k) = \sum_{i=0}^k (-1)^i \binom{n}{i}\binom{n-i}{k-i}^m$$
yields an interesting Binomial identity. A special case occurs when $n=aN$ is a multiple of $a=2,3,4,$ etc; $k=(a-1)N+1,$ and $m=a,$ for then
$$km = ((a-1)N+1)a = a(a-1)N + a \gt a(a-1)N = n(m-1).$$
The identity is
$$0 = f(aN, a, (a-1)N+1)=\sum_{i=0}^{(a-1)N+1} (-1)^i \binom{aN}{i}\binom{aN-i}{(a-1)N+1-i}^a.$$