2
$\begingroup$

Suppose you have a new algorithm that you want to publish. Are there any best practices and methodologies you usually consider in order to test the robustness and performance of a new method?

The case is a novel classification algorithm of matrices composed by observations x features (e.g. a gene expression matrix).

$\endgroup$
  • $\begingroup$ You will of course want to compare your classifier to its competitors. Since you say that the classifier is novel, I presume that you have a good knowledge of which comparable classifiers exist. One idea could be to have a look at the papers in which those classifiers were proposed, and to let the methodology used therein guide your own comparison. $\endgroup$ – MånsT May 13 '14 at 11:01
  • $\begingroup$ Hi @MånsT. Thanks for your suggestion. Of course, it is a good idea. I was thinking about a standard de facto on this field, and why it could be useful. $\endgroup$ – gc5 May 15 '14 at 10:16
2
$\begingroup$

Test it on several known datasets and use them as benchmarks. Such as the iris dataset http://en.wikipedia.org/wiki/Iris_flower_data_set or whatever is the common dataset in that field.

You would want to compare it in both speed and accuracy.

$\endgroup$
2
$\begingroup$

Outlier detection and robustness are two ends of the same stick.

In robustness your objective is to fit a model on contaminated data such that you find a fit as close as possible as the one you would have had without the outliers.

Outlier detection tries to find all the outliers that matter in the sample. That is, all points that exert a disproportional pull on the fitted parameter of the model.

If your algorithm is equivariant in some sense (the exact sense depends on the estimation problem, for example for regression a natural group of equivariance is regression equivariance, for classification, a natural group of equivariance for the vector of fitted probabilities is affine equivariance), then, in many cases the worst possible (or adversary) configuration of outliers, in the sense of being those that are hardest to find, are either known or can be worked out with reasonable confidence.

Then, you should simulate from and test your procedure preliminary on those (adversary) cases. The bias (or difference between the fit you have on contaminated data and the fit you would have had without the outliers) computed on adversary configuration of outliers is how you want to measure the performance of any anomaly detection procedure.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.