You can't know in advance which clustering algorithm would be better, but there are some clues, for example if you want to cluster images there are certain algorithms you should try first like Fuzzy Art, or if you want to group faces you should start with (GGCI) global geometric clustering for image.
Anyway this does not guarantee the best result, so what I would do is use a program that allows you to methodically run different cluster algorithms, such as weka, RapidMiner or even R (which is non visual), There I will set the program to launch all the different clustering algorithms I can, with all the possible different distances, and if they need parameters, experiment each one with a variety of different parameter values (besides if I do not know the amount of clusters, run each one with a variety of numbers of it). Once you settled the experiment, leave it running, but remember to store somewhere the results of each clustering run.
Then compare the results in order to obtain the best resulting clustering. This is tricky because there are several metrics you can compare and not all are provided by every algorithm. For example fuzzy clustering algorithms have different metrics than non-fuzzy, but they can still be compared by considering the fuzzy resulting groups as non-fuzzy,
I will stick for the comparison to the classic metrics such as:
• SSE: sum of the square error from the items of each cluster.
• Inter cluster distance: sum of the square distance between each cluster centroid.
• Intra cluster distance for each cluster: sum of the square distance from the items of each cluster to its centroid.
• Maximum Radius: largest distance from an instance to its cluster centroid.
• Average Radius: sum of the largest distance from an instance to its cluster centroid divided by the number of clusters.