Which clustering algorithm did you try?
k-means is known to not work very well with noise. Hierarchical clustering is very likely to produce single-element clusters. That's outliers, nothing wrong with that. Or you might try DBSCAN, in which the "N" stands for "Noise". That algorithm actually is designed to be able to handle some noise objects. You can look up details on WikipediaWikipedia.