# Clustering sets of vectors

I have a set of $d$-dimensional vectors $\{v_1,v_2,\dots,v_n\}$, each of which has been assigned a label from a set $S=\{s_1,s_2,\dots,s_k\}$. I would like to find another set of labels $T=\{t_1,t_2,\dots,t_l\}$ where $l < k$, such that all vectors having the same $S$ label also have the same $T$ label. In other words $T$ is a strictly coarser clustering than $S$. My question is, what is a good way to go about finding this $T$ clustering?

The obvious approach would be to take the mean of all of the vectors having a given S label, and then cluster these new $s$ vectors. However I feel like this throws away a lot of potentially useful information about the distribution of the vectors that went into computing those means. Is there another method for finding this $T$ clustering which makes better use of the $v$ vectors? Thanks in advance.