Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have multiclass unbalanced data (4 class with 15% 25% 45% 15% data in each class). Which method is good for classification of such data- SVM or ANN?

UPDATE- Let me make the question little more general. @Dikran Marsupial said in one answer "choice of classifier depends on the nature of the particular dataset" but what are the factors that one should consider before choosing a classifier. I understand the first chose may not give best answer all the time but it can be a good starting point. So what properties of data I should consider before choosing a classifier??

share|improve this question
    
Please provide more info about this data. –  mbq Mar 12 '12 at 18:34
    
the data have 25 dimensions and about 2000 data points. –  d.putto Mar 13 '12 at 10:32
add comment

3 Answers 3

up vote 10 down vote accepted

The no-free lunch theorems suggest there is no classifier that is a-priori superior to any other, and the choice of classifier depends on the nature of the particular dataset. I wouldn't comit myself to a choice of classifier and would instead evaluate several methods.

The classes are only mildly imbalanced, so I suspect that shouldn't be a key factor in the decision of which classifier to use.

A more important question would be whether you wanted a simple discrete classification, or whiether you wanted estimates of the probabilities of class membership, for examples because you have unknown or variable mislcassification costs, or relative class frequencies, or if it would be beneficial to have a "reject" option. In that case the SVM is not a good choice as it is designed for discrete classification, and rather than post-processing the output to get probabilities it is better to use a method that was designed to provide a probabilistic output in the first place, such as kernel logistic regression.

share|improve this answer
    
Let me make the question little more general. As you said "choice of classifier depends on the nature of the particular dataset" but what are the factors that one should consider before choosing a classifier. I understand the first chose may not give best answer all the time but it can be a good starting point. So what properties of data I should consider before choosing a classifier?? –  d.putto Mar 14 '12 at 9:55
2  
Good questions to ask are (i) whether a probabilistic output is required (ii) is the dataset large or small (relative to the number of attributes), i.e. is the chief problem avoiding over-fitting, or in dealing with the volume of data (iii) are the classes imbalanced (iv) are the attributes discrete or continuous or both (v) do I have any expert knowledge that I can build into the model (in which case you need to have a model that can accommodate that knowledge) (vi) are there missing data (vii) can the labels be incorrect sometimes (viii) are the misclassification costs equal. –  Dikran Marsupial Mar 14 '12 at 10:43
1  
(ix) are the training set class frequencies the same as the operational conditions (x) are the training data really an i.i.d. same from the true joint distribution. These are a few of the kinds of questions that may affect practical applications. However, even then it can be difficult to choose the right classifier without actually trying them out to see which works best. Fully automated pattern recognition is something I have been working on and it is a difficult problem. –  Dikran Marsupial Mar 14 '12 at 10:46
    
I am looking for probabilistic output and wanted to know more about "kernel logistic regression". If possible, please indicate some link/tutorial related to that. (Note- I am not a mathematician/statistician and I will prefer something with less theory more practical applications example) –  d.putto May 4 '12 at 8:35
    
if you use MATLAB, then my GKM toolbox (which includes kernel logistic regression) is available here theoval.cmp.uea.ac.uk/projects/gkm . Sadly there is no manual yet, just the conference paper, but there are some demo programs that will at leat get you started. I don't know of any more accessible introduction than my paper, which should be O.K. for anyone that understands GLMs. –  Dikran Marsupial May 4 '12 at 9:23
add comment

For specificity in the following I'm going to assume that an ANN here means a feedforward multilayer neural network / perceptron as discussed in e.g. Bishop 1996. and an SVM is the the vanilla version e.g. from Hastie and Tibshirani.

@Dikran Marsupial's points about the structure of the domain are important ones. In fact you might want to read DM's other answer about SVMs. The possibility of having a posterior over classes is important if you expect to apply a loss function or otherwise act on your level of classification certainty as well as the actual classification. If not: well, not.

In addition, I can see four more ways to choose.

Loss function

One way to distinguish the two is to decide whose loss function you prefer. Classically, ANNs have smooth loss functions, e.g. cross-entropy for multi-class classification. SVMs tend to have some kind of 'hinge loss': 0 to a point then increasing. One of these may be a more natural fit to your problem.

Data size

Another consideration is data size and storage. You mention your category balance but not the total size of the data. SVMs by definition keep and use only the 'support vectors', a subset of observations that anchor the separating hyperplane(s). This can make for a small final classifier. Also, traditional ANN training can be slow - the space of functions as smooth as the implicit gaussian process that your ANN is approximating with its finite number of hidden nodes is large...

Multiple classes

If you have multi-category data, SVMs have several ways to construct the necessary multi-class classifier out of individual two class SVM models. At least three methods are available which, as @fabee points out, may not give the same answers. His reference looks like a useful one. The options are a lot clearer in ordinary smoothed statistical classification model territory, where your ANN belongs.

Interpretability

If you care about discerning the the importance of different covariates, then ANNs give you hyperparameters to do so, although more traditional methods might be as or more efficient and straightforward at this, e.g. the Lasso (L1 regularisation) for linear regression models. If prediction success is your only goal then this aspect is, of course, irrelevant.

share|improve this answer
1  
The sparsity of SVMs is often rather limited, in my experience, if you choose the hyper-parameters to minimise the cross-validation error, you often end up with an SVM where almost everything is an SV. If you need sparsity, it is better to use an algorithm where sparsity was a deliberate aim, rather than a happy accident. Also it is possible to get a sparse ANN model, I used to use the Laplace prior introduced by Peter Williams quite a lot and it generally works fairly well see mitpressjournals.org/doi/abs/10.1162/neco.1995.7.1.117 . –  Dikran Marsupial Mar 12 '12 at 19:09
    
Also I have found that using ARD for feature selection using evidence minimisation for ANNs and cross-validation for kernel models is very prone to over-fitting and more often than not makes generalisation performance worse. (+1 by the way) –  Dikran Marsupial Mar 12 '12 at 19:10
add comment

This question cannot be answered generically. It even depends on the multi-class classification strategy that you are using (i.e. one-vs-one, one-vs-rest, ...). Personally, I would use an SVM and choose an multi-class strategy that fit my problem and my computational resources. A nice paper how to do that is:

Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers Erin L Allwein, Robert E Schapire, Yoram Singer in Journal of Machine Learning Research (2001)

If you want every class of your dataset to be equally important you can either use the quick and dirty hack of cloning datapoints in the smaller classes until each class has the same number of datapoints or you could use an SVM implementation that allows you to set different penalization constants C for each class.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.