How can an artificial neural network ANN, be used for unsupervised clustering? I understand how an artificial neural network (ANN), can be trained in a supervised manner using backpropogation to improve the fitting by decreasing the error in the predictions. I have heard that an ANN can be used for unsupervised learning but how can this be done without a cost function of some sort to guide the optimization stages? With k-means or the EM algorithm there is a function for which each iteration searches to increase. 


*

*How can we do clustering with an ANN and what mechanism does it use
to group data points in the same locality?


(and what extra capabilities are brought with adding more layers to it?)
 A: Neural networks are widely used in unsupervised learning in order to learn better representations of the input data. For example, given a set of text documents, NN can learn a mapping from document to real-valued vector in such a way that resulting vectors are similar for documents with similar content, i.e. distance preserving. This can be achieved using, for example, auto-encoders - a model that is trained to reconstruct the original vector from a smaller representation (hidden layer activations) with reconstruction error (distance from the ID function) as cost function. This process doesn't give you clusters, but  it creates meaningful representations that can be used for clustering. You could, for instance, run a clustering algorithm on the hidden layer's activations.
Clustering: There are a number of different NN architectures specifically designed for clustering. The most widely known is probably self organizing maps. A SOM is a NN that has a set of neurons connected to form a topological grid (usually rectangular). When some pattern is presented to an SOM, the neuron with closest weight vector is considered a winner and its weights are adapted to the pattern, as well as the weights of its neighbourhood. In this way an SOM naturally finds data clusters. A somewhat related algorithm is growing neural gas (it is not limited to predefined number of neurons). 
Another approach is Adaptive Resonance Theory where we have two layers: "comparison field" and "recognition field". Recognition field also determines the best match (neuron) to the vector transferred from the comparison field and also have lateral inhibitory connections. Implementation details and exact equations can readily found by googling the names of these models, so I won't put them here.
A: You want to look into self-organizing maps.  Kohonen (who invented them) wrote a book about them.  There are packages for this in R (som, kohonen), and there are implementations in other languages such as MATLAB.  
A: Maybe the Clustering with Neural Network and Index (CNNI) model is what you are looking for.
https://doi.org/10.31219/osf.io/ejxm6
CNNI uses a Neural Network to cluster data points. Training of the Neural Network mimics supervised learning, with an internal clustering evaluation index acting as the loss function. It successively adjusts the weights of the Neural Network to reduce the loss (improve the value of the index).
Structure of CNNI:
The structure of CNNI is simple: a Neural Network for supervised learning plus an internal clustering evaluation index. The index acts as the loss function, because there is no target output associated with each input data point in clustering scenario.
The number of neurons in the input layer of CNNI equals to the dimension of the data points given to the network. The number of neurons in the output layer of CNNI equals to $ K $ (number of clusters we want to classify). By comparing the values of each output neuron, label of one data point is obtained (e.g., find out the maximum of output neurons).
Training of CNNI has some difference from other supervised learning Neural Networks. We need to compute each data point's label according to the Neural Network's current state, then calculate the value of the clustering evaluation index, according to the labels of all data points.  Adjustment of the weights of the Neural Network is based on the value of the index.
An experiment shows CNNI equipped with MMJ-SC, achieves the first parametric (inductive) clustering model that can deal with non-convex shaped (non-flat geometry) data, which implies it is the first general-purpose parametric (inductive) clustering model.
