Clustering and blind signal separation (BSS) are often used together in an application, and when this is the case, the BSS algorithm comes first as a pre-processing step in order to "reduce the dimension of the problem". The original inputs can then be accordingly "cut down" before being fed into a clustering algorithm to optimally segment the now lower order problem. Because the dimensions have been reduced, the result of clustering can now be readily visualised in 2 or 3 dimensions.
In "input-[process]-output" form, we have a linked, three-step filter:
input (high dimensional, mixed source) -> [Blind signal separation] ...
-> ranking of features -> [Dimension reduction] -> lower dimensional inputs ...
-> [Clustering] -> optimal segmentation.
Suppose that your inputs are vectors, i.e. each of your data points / samples has a number of attributes, say $n$ of them.
In simple terms, clustering takes the inputs, considers them in an $n$-dimensional space, and, given a target number of clusters, runs a mathematical algorithm to decide what should be the centre of each cluster and which points should be assigned to belong to which cluster.
So clustering is essentially mathematical segmentation of your data into groups (optimal segmentation if you will).
But the challenge with using clustering on your raw vector inputs is that the algorithm is having to work in $n$ dimensional space -- which means its difficult to visualise, and, if many of the attributes are correlated, then those extra dimensions are not adding much value in the problem of identifying the best clusters.
Enter blind source separation...
Blind Source Separation:
Blind signal separation (BSS), on the other hand, is about separating a mixture into individual components.
Again, in simple terms, suppose that you have a process that "mixes" or confounds a number of pure signals into an aggregate whole. As an example, think of taking a recording from a number of microphones situated in an orchestra hall where there are a number of instruments all playing the same melody but where there is also quite a bit of local chattering among the audience. The resulting recording is a mixture of all of this.
The question in this case is, from the mixed input, and without knowing how the mixture is composed, can you take the output (recording) and separate out the individual input vectors?
So BSS is essentially an inverse problem in which you start with a mixed input and attempt to separate out the individual elements that went into the mixing process.
BSS, Dimension Reduction, and Clustering:
I mentioned at the start that clustering and BSS are often used together. The reason for that brings in the concept of dimensionality reduction.
The input into BSS consists of mixed signals plus noise (uncorrelated, white noise for example, or low correlated sources of little interest).
BSS works by identifying, from a number of 'features' about the signals (mathematical expressions involving the individual attributes of each vector), those features that 'explain' the greatest variation in the data.
These features can then be ranked in descending order. By taking the top three features, for example, one arrives at a much more manageable number of dimensions in which to perform clustering.
A typical example and real-world application:
So in a typical example, one might first apply PCA (principal components analysis) -- which is a type of BSS algorithm -- to a data set to "discover" the top 3 features that are most useful in explaining the variation in the data set, and then use mathematical clustering on just those 3 features to identify the segments into which the data can be split.
I've seen this combination approach used very successfully in the problem of classification (unsupervised learning) of bottom-oriented sonar signals to determine automatically the type of sea bottom that a vessel is travelling over: is it muddy, sandy, rocky, without having to send a diver down to check.
So, when used in combination, these techniques can become quite powerful tools.