I observe $N$ observations $\{x_{1,t_1}, \dots, x_{N,t_N}\}$ from a $k$ component Gaussian Mixture model. The $i$th observation is seen at time stamp $t_i$ and is distributed such that each $x_{i,t_i}|\boldsymbol{\pi}, \boldsymbol{\mu} \sim \sum_{j=1}^{k} \pi_j \mathcal{N}(\mu_j, \sigma_j)$. However, it is true that each observation with the same time stamp $t_i$ must (each) come from different components. Therefore, if I observe $x_{1,1}, x_{2,1}, x_{3,1}$ at time stamp $1$, then I know that each of the $3$ observations come from 3 different components and also that $k \geq 3$.
Now say I have a Gibbs sampler, and I wish to in one of the steps sample the group/allocation label of each observation to its group $z_i \in \{1, \dots k\}$. In a standard Gibbs sampler, this can be done by putting a dirichlet prior over the mixing weights and sampling the allocation of each observation $z_i$ to each of the $j = \{1, \dots, k\}$ components with probability proportional to $\pi_j \exp \left(\frac{(x_{i,t_i} - \mu_j)^2}{2\sigma_j^2} \right).$ However, I cannot directly use this now because observations are not time dependent. In particular, if at one MCMC iteration, $x_{1,1}$ has already been allocated cluster $2$, then the conditional probability of allocating, say, $x_{2,1}$ to cluster $2$ will be zero, but it still able to take this class.
One idea I had was to block sample all observations seen at the same time stamp, however, I realise this may get computationally heavy if the number of observations seen at any one time is large, and $k$ is also large. Does anyone know how this can be done? Thanks.