If Manhattan distance always performs better on a dataset...what does it mean? I'm analyzing my dataset using kNN. I experimented with various distance functions but Manhattan seems to perform better in terms of lowest RMSE over various values of k. 
I've read a bit about Manhattan and how it differs from Euclidian but I can't seem to answer the question "If Manhattan performs better on a particular dataset, do we find out something about the property of the dataset that we wouldn't know otherwise" or in other words "Is manhattan usually a good distance function for particular type of datasets"?
In my example, I'm using the wine quality dataset where I look at various chemical properties of wines and try to predict their rating (between 3 and 9)
 A: Also use the search terms l1 norm, l1 distance, absolute deviance etc all of which refer to the same thing as manhattan distance.
The properties of the l1-norm (manhattan distance) can largely be deduced from its shape (ie it is V shaped instead of U shaped like the parabola of the l2-norm (euclidian distance). The l1-norm  can be said to be less sensitive to outliers and more sensitive to small scale behavior then the l2-norm.
Ie it will tend to "drive things to zero" focusing on small scale behavior because it doesn't flatten out around zero like a parabola. It will also be less sensitive to large distances because the slope does not increase with distance from the origin. This can result in a model that fits part of the data very well/exactly but ignores a few dimensions or cases that don't fit with the rest of the data.
I suspect that these properties explain its performance on the data set you are seeing. Ie in this classification problem it is better to have an exact/excellent match on a few of the dimensions and miss some of the other dimensions then to do fairly well on all of the dimensions.
These reasons also explain why the l1-norm is often used in robust regression (where it will ignore outliers) or as a penalty in the lasso algorithm (where it will drive some of the coefficients to zero resulting in a simpler model). 
