Evaluating features and similarity measures I am currently developing a classifier, which is supposed to classify into a number of classes. For this purpose I am
designing some features and similarity measures which I might use for a later kernel for a support vector machine (SVM).
Now I was thinking to myself, how I can roughly evaluate my features and the similarity measures without having to develop
the whole classifier first.
Here are my ideas, and I'd be very pleased if you could tell me, if they are valid or not (and maybe also give papers where
it was done similarly..?)


*

*To roughly evaluate the features, I could make a Naive Bayes assumption and try to use one feature at a time with a Naive Bayesian (NB) classifier
and see how it performs.

*To evaluate the similarity measures (and at the same time features) I could calculate the inter-class mean similarity
and intra-class mean similarity and see how far it differs. I was thinking, that I could maybe use an ANOVA test to see, if
the difference is significant. However, can I just assume that the results are normally distributed and have the
same variance? And ANOVA would simply tell me, that one mean is significantly different but it doesn't tell me which one. Maybe
another statistical test is more suitable?
 A: You could make the (very naive) assumption that the best features for SVM will also be the best ones for other classifiers as NB. Therefore you can (as you said) select the best features for NB. You can also include the similarity ones in this set.
Other (better) option is to apply feature selection weighting to see which features separate the classes better. In Text Classification, chi^2 is commonly applied and it will provide an idea (and a ranking) of how good the features are.
One question though, why do you want to select the features in advance in this fashion? I mean, apart from scalability issues, SVM is known for dealing well with very large number of features.
I think I have only partially answer your question, I hope it helps...
A: Using one feature at a time would cancel out the multivariate advantage. You might loose important features for your classification problem because taken independently they do not show  clear differences.
To reduce this issue you could construct the Naive Bayes classifier on subgroups of features (randomly selected) but this might take more time than constructing your final classifier.
A: I would create the similarity (distance) matrix of size N*N for your dataset of N observations with M features. The M features all feed into the pairwise similarity/distance scores in your N*N matrix.
Then feed this (similarity) matrix into an agglomerative or hierarchical clustering algorithm and look at the cluster result (the dendrogram) to see how the observation pairs clump up together and what would be a reasonable cut-off value (tree height) to form your clusters.
Also, I would feed in the similarity matrix into an MDS algorithm (my favorite is ISOMAP), and then output the results in a scatter plot. You can visually assess if your similarity measure is good (or not) by looking at this plot easily. For instance, are you certain that observation A and B are very similar? Then, you should see them close to each other in 2-D. The projected distances on this plot will be approximate, but should be good enough for you to assess whether your similarity measure makes sense.
You can further colour-label the observations in your scatter plot with respect to their cluster labels from your hierarchical clustering exercise. You should see that observations from the same clusters clump up nicely in the 2-D scatter plot. 
Hierarchical clustering package in R.
MDS and ISOMAP algorithms in R.
