General practice for clustering is to do some sort of linear/non-linear dimensionality reduction before clustering esp. if the number the number of features are high(say n). In case of linear dimensionality reduction technique like PCA, the objective is to find principal orthogonal components(say m) that can explain most variance in the data such that m<<n when n is high.

But for non-linear dimensionality reduction techniques like auto-encoders, can the reduced dimensions, itself be clusters that indicate different modes of operation example for industrial components. Am I missing something here or is my understanding of non-linear dimensionally reduction wrong? Any help is appreciated.

This question might be too basic for some, so please don't be extremely critical of the question if you don't want to answer it.

@fk128 shared his interpretation of my question that might be better understood and easy to interpret than what I have mentioned above

  • 2
    $\begingroup$ "General practice" says who? I disagree. I avoid this whenever possible. $\endgroup$ – Has QUIT--Anony-Mousse Apr 29 '18 at 6:37
  • $\begingroup$ Most of the research papers and even the package creators for example hdbscan recommends dimensionality reduction before applying clustering esp. If the number of dimensions are more than 50. How do you do clustering when the number of dimensions are very high with out dimensionality reduction ? $\endgroup$ – codecodercoding Apr 29 '18 at 6:40
  • $\begingroup$ If you have such data, the axes will likely have very different meaning. You cannot reliably weight them. So whatever you do (PCA or not) it is statistically pretty much "black magic" aka nonsense. The result will not have any mathematical properties. $\endgroup$ – Has QUIT--Anony-Mousse Apr 29 '18 at 6:44
  • $\begingroup$ Not everything that people do is correct or sensible. It's often just to get "some" output because their boss wants some output.and when nobody can verify the quality, it cannot be wrong. But to be of scientific value, you need to be able to test and falsify things... but you can't do this anymore. $\endgroup$ – Has QUIT--Anony-Mousse Apr 29 '18 at 6:47

There are two types of clustering: hard and soft. Hard is when you assign a specific data point to a single cluster/category. So if there are $k$ clusters, a data point $x$ can only be assigned to one cluster in $\{1,..., k\}$. Soft/fuzzy is when a data point is assigned to multiple clusters, and the assignment is represented with membership weights, such that all the weights add up to 1. So for example if there are 3 clusters, a datapoint $x$ can have weights $w = [0.1,0.6,0.3]$ such that $0.1 + 0.6 + 0.3 = 1$. You can also use the weights representation in the hard clustering case, so you'd only have $w = [1, 0, 0]$, where one component is 1 and the rest are zero (this is also referred to as one-hot encoding).

Any sort of dimensionality reduction, linear or non-linear, reduces the dimensions of the input features from $n$ to say $m < n$. So if you have a data point $x$ with dimension $n$, it is transformed into a data point $x'$ with dimension $m < n$.

Assuming I understand your question correctly, your question is whether with an auto-encoder the components of $x'$ can represent cluster membership weights, i.e. $x' = w$. However, this is extremely unlikely to be the case.

Typically, to obtain clusters, you can later run any clustering algorithm on the dimensionally reduced data, regardless whether it was obtained from a linear or non-linear dimensionality reduction technique.

| cite | improve this answer | |
  • $\begingroup$ Your interpretation is correct and is much better at explaining my question than what I have written above. So I have made an edit to look at your response for better understanding. Thanks for the answer. Is there any intuitive or theoretical understanding of why these results has the potential to be different. Since neural networks(NN) has the potential to be good universal approximatiors, shouldn't they ideally give same result if the latent space embedding is really good ? $\endgroup$ – codecodercoding Apr 29 '18 at 7:23
  • $\begingroup$ Clustering and dimensionality reduction are two different things. By analogy, you can think of a supervised learning task like classification as 'supervised clustering' as the goal is to predict a category for each input. This task is really difficult to do in an unsupervised way, which is why the state-of-the-art methods are all supervised. However, there are some NN methods that could be thought of as doing clustering, e.g. Self-Organising Maps, see sciencedirect.com/science/article/pii/S089360800900207X $\endgroup$ – fk128 Apr 29 '18 at 11:08

The components of an autoencoder are supposedly even less reliable than your usual clustering.

Why don't you just try it: train autoencoders on some data sets, and visualize the "clusters" you get from the components?

While this great answer on tSNE for clustering is specific for tSNE, I believe the results for other such encoders will be similar: they will cause fake clusters because of emphasizing some random fluctuations in data.

| cite | improve this answer | |
  • $\begingroup$ Can you clarify on what do you mean by reliable in unsupervised learning ? $\endgroup$ – codecodercoding Apr 29 '18 at 6:41
  • $\begingroup$ Clustering is never very reliable. $\endgroup$ – Has QUIT--Anony-Mousse Apr 29 '18 at 6:42
  • $\begingroup$ Thanks for sharing the link to great answer. I can understand the problem with tSNE as it doesn't preserve either density or distance from original space. Does all the dimesnionalty reduction techniques follow logic. UMAP is a recent dimensionality reduction technique that is hypothesised to be a good precursor to density based clustering. Have you checked that out yet? $\endgroup$ – codecodercoding Apr 29 '18 at 6:50
  • $\begingroup$ PCA (but without using the eigenvalues for scaling!) and random projections (c.f. Johnson Lindenstrass lemma) are supposed to preserve density optimally. But I do not see any benefit of using it. Never heard of UMAP, so I cannot comment on it. Beware that these drawbacks of tSNE appear to be relatively new insights, not in the original publication. So are there any independent evaluations of UMAP wrt. this feature? $\endgroup$ – Has QUIT--Anony-Mousse Apr 29 '18 at 9:36
  • $\begingroup$ I am currently using UMAP on my dataset, but I am failing to say how well it performed since I am doing unsupervised learning on large data sets. I have few events that happened during the timeline of the data. I am using analysis of anomalies and clusters formed before the event, to understand how well the stack of methods that I performed are working together as a whole. $\endgroup$ – codecodercoding May 1 '18 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.