What if I have a method which performs worse on the development but better on the test set? Assume I have a baseline system and a method proposed by me, and I want to check whether my method is better than the baseline system or not.
I use both of them to train on the same training set and tune the hyper-parameters behind each method (the regularization term for example) on the same development set, and I choose the best models for the baseline and my method separately on the development set and evaluate them on the same test set now.
It is quite common that a method might perform well on the development set compared to the baseline, but worse than the baseline on the test set, and we can say that this method might overfit on the development set.
However, what if it works the other way that my method performs worse than the baseline on the development set but better than the baseline on the test set. Does it mean the model generalization is good, or can we draw some useful conclusions here?
 A: Assuming that the test set and the dev set are independently and identically sampled from some fixed distribution, the performance on each set should be the same plus some noise. When you pick the point that has the best dev set performance then you probably picked one where the "noise" is favorable to you. It makes sense for the test set accuracy to be less than on dev because of regression to the mean. 
The level of noise is related to the size of the sample. If your sample is small then the noise can have a bigger effect. It would not be too surprising for the test set to occasionally do better than the dev. If the sample is big then the performance should be similar on both.
If the performance on the test set is significantly better then the dev set then it is evidence that the partitioning of the data was not random. Maybe there is some structural reason for why the test set is easier like the distribution of labels is different or it contains data selected from a different time period.
