Internal evaluation metrics are not very comparable across different data sets (neither across different projections and other preprocessing). So for fairness, you should indeed use the same distance matrix when computing Silhouette.
At the same time, it is to be expected that the results suffer when not using the original data. PCA rescales the data, making other directions more important. That of course means that a solution found in this rescaled version is not optimized for the original data.
The original data may be the wrong place to compute distances. Then you shouldn't have been clustering there in the first place. First identify the best projection to evaluate distances, then proceed. If distances do not work, Silhouette is meaningless. It always boils down to this question: how to compute meaningful distances. Before you solve this, you can't cluster not evaluate.
Because of this, one may also argue to use Silhouette in the projected space. But as mentioned above, such a projection usually makes the problem easier (by dimensionality reduction) and hence one would expect the Silhouette scores to go up.
Silhouette is still one of the better cases. SSQ values would be completely incomparable!
So either way will not allow you to make a sound statement of whether the result is better with PCA or without. It is always apples with oranges. Thus, I would avoid making any comparative statements! It's just not sound, no matter how you do it.
Instead, I recommend to only draw the following conclusions:
- if all Silhouette scores are below 0.2, do not draw any conclusions from them. It's probably the data, not the algorithm, that doesn't work.
- if any Silhouette is larger than 0.5 then it is likely a good result
- if any Silhouette is larger than 0.7 then it is likely a very good result
- if any Silhouette is larger than 0.9 then there is something wrong
- if the projected algorithm scores better in Silhouette on the original data, then the projection definitely was a good idea (but the converse does not hold)
- if the original clustering scores bettet-or-similar in Silhouette on the PCA-output, then PCA was not necessary (again, the converse does not hold)