Distance measure methods of R function dist() evaluation I want to compute the distance matrix for the columns on a 1000 x 230 matrix using the dist() function in R. Though, I am uncertain about which method to use.
I know the differences between the methods and how the algorithm works, but I would like to hear from you which one will you prefer when working on gene expression data sets. The values are z-scores derived from normalized gene expression data. That is, data are real-valued from -10 to 10 (roughly) and it is important if the value is negative or positive (means: if the gene is upregulated or downregulated).
Could be rephrased to make it more specific: Which of the methods dist() function supports could be the optimal to understand the differences and similarities for the columns of my matrix. 
 A: Well, lets work through this one-by-one. 


*

*Minkowski norms with $p<1$ are not true distance metrics, so those are out provided you wish to use a true distance metric. This leaves $L_p$ norms for $p\ge1.$

*Euclidean ($L_2$) isn't great in high dimensions.

*As $p$ gets smaller, it is less terrible in the sense of the curse of dimensional sense, so Manhattan ($L_1$) is a popular choice. That is, it tends not to be dominated by the dimension in which the difference between the two points is largest.

*Likewise, the above observation excludes the $L_\infty$ norm. 

*Canberra is intended for nonnegative values, e.g. counts; your data may be negative, so it's excluded. 

*Binary is only defined for binary data, so it is excluded as well.

A: You can forget about Canberra if your data assumes negative values.
But the distance to use depends on the purpose!
Are you doing prediction from your data? Then, you don't need an expert advice, cross-validating carefully over the distances will give you the distance that helps you achieve highest performance on your model.
Are you trying to visualize clusters the data? Then, once you obtained the distance matrix, plot the first axis of a Multidimensional Anlaysis (R has a lot of packages for this)
Are you trying to describe the data? Then you need to know the differences between every distance, and bear in mind that the conclusions you obtain will highly depend on the data. Euclidean metric may overweight large differences...
A: I take it you work with fold changes, or log fold changes, hence the negative numbers. You could switch to positive ratios instead and use Canberra. I like it because it measures the relative distance, as opposed to absolute distance in Euclidean and such like. 
