I understand how an artificial neural network (ANN), can be trained in a supervised manner using backpropogation to improve the fitting by decreasing the error in the predictions. I have heard that an ANN can be used for unsupervised learning but how can this be done without a cost function of some sort to guide the optimization stages? With k-means or the EM algorithm there is a function for which each iteration searches to increase.

  • How can we do clustering with an ANN and what mechanism does it use to group data points in the same locality?

(and what extra capabilities are brought with adding more layers to it?)

  • Are you interested in NN unsupervised learning in general, or specifically in unsupervised clustering with neural networks? – Denis Tarasov Mar 3 '15 at 16:44
  • @DenisTarasov, I am interested primarily in unsupervised clustering with NN, but do not know much about NN unsupervised NN learning in general. It would be great if an answer would include a bit of the NN unsupervised learning in general before discussing the specific application. – Vass Mar 3 '15 at 17:02
  • 1
    Self Organizing Map (SOM) is a type of network used for clustering. – Cowboy Trader Mar 3 '15 at 17:16
  • unsupervised learning in ANN – It extracts statistical properties from the training set. – Unsupervised learning is more difficult but is seen as biologically plausible - Requires no teacher. – yonas Apr 3 '17 at 1:29
up vote 54 down vote accepted

Neural networks are widely used in unsupervised learning in order to learn better representations of the input data. For example, given a set of text documents, NN can learn a mapping from document to real-valued vector in such a way that resulting vectors are similar for documents with similar content, i.e. distance preserving. This can be achieved using, for example, auto-encoders - a model that is trained to reconstruct the original vector from a smaller representation (hidden layer activations) with reconstruction error (distance from the ID function) as cost function. This process doesn't give you clusters, but it creates meaningful representations that can be used for clustering. You could, for instance, run a clustering algorithm on the hidden layer's activations.

Clustering: There are a number of different NN architectures specifically designed for clustering. The most widely known is probably self organizing maps. A SOM is a NN that has a set of neurons connected to form a topological grid (usually rectangular). When some pattern is presented to an SOM, the neuron with closest weight vector is considered a winner and its weights are adapted to the pattern, as well as the weights of its neighbourhood. In this way an SOM naturally finds data clusters. A somewhat related algorithm is growing neural gas (it is not limited to predefined number of neurons).

Another approach is Adaptive Resonance Theory where we have two layers: "comparison field" and "recognition field". Recognition field also determines the best match (neuron) to the vector transferred from the comparison field and also have lateral inhibitory connections. Implementation details and exact equations can readily found by googling the names of these models, so I won't put them here.

You want to look into self-organizing maps. Kohonen (who invented them) wrote a book about them. There are packages for this in R (som, kohonen), and there are implementations in other languages such as MATLAB.

  • can you go into some detail about how the NN can do this and elaborate on the theory? possibly also explain the effect of using a deep NN (DNN)? – Vass Mar 3 '15 at 17:22
  • 1
    I'm afraid I have very little expertise here, @Vass. I don't think adding extra layers will do much, other than slow it down. Someone else will have to give you the theory, I just wanted to get you started. – gung Mar 3 '15 at 17:25
  • 1
    A student in our lab experimented with clustering using SOM. It took forever to run and the results were very disappointing compared to other approaches (in our case standard graph clustering algorithms). I have always been puzzled by the fact that the standard 2D target domain (topological grid) seems to be a highly arbitrary space. More worryingly is is very simplistic and essentially needs to compress the data into a space described by just two variables. – micans Mar 4 '15 at 11:39
  • @micans makes some good points, but the data is not simply compressed into a space described by just two variables, because each node is also associated with a prototype. Also if the running is slow, it may well be an implementation issue. Statistically, other methods than SOM should achieve better classification results. As for the topology issue, the brain seems to be organised as layers of 2D topology, but it achieves great results (or so I'd like to think). – Tom Anderson May 8 '17 at 3:30

protected by Glen_b Apr 3 '17 at 5:19

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.