During an experiment for text classification, I found ridge classifier generating results that constantly top the tests among those classifiers that are more commonly mentioned and applied for text mining tasks, such as SVM, NB, kNN, etc. Though, I haven't elaborated on optimizing each classifier on this specific text classification task except some simple tweaks about parameters.

Such result was also mentioned Dikran Marsupial.

Not coming from statistics background, after read through some materials online, I still cannot figure out the main reasons for this. Could anyone provide some insights on such outcome?


Text classification problems tend to be quite high dimensional (many features), and high dimensional problems are likely to be linearly separable (as you can separate any d+1 points in a d-dimensional space with a linear classifier, regardless of how the points are labelled). So linear classifiers, whether ridge regression or SVM with a linear kernel, are likely to do well. In both cases, the ridge parameter or C for the SVM (as tdc mentions +1) control the complexity of the classifier and help to avoid over-fitting by separating the patterns of each class by large margins (i.e. the decision surface passes down the middle of the gap between the two collections of points). However to get good performance the ridge/regularisation parameters need to be properly tuned (I use leave-one-out cross-validation as it is cheap).

However, the reason that ridge regression works well is that non-linear methods are too powerful and it is difficult to avoid over-fitting. There may be a non-linear classifier that gives better generalisation performance than the best linear model, but it is too difficult to estimate those parameters using the finite sample of training data that we have. In practice, the simpler the model, the less problem we have in estimating the parameters, so there is less tendency to over-fit, so we get better results in practice.

Another issues is feature selection, ridge regression avoids over-fitting by regularising the weights to keep them small, and model selection is straight forward as you only have to choose the value of a single regression parameter. If you try to avoid over-fitting by picking the optimal set of features, then model selection becomes difficult as there is a degree of freedom (sort of) for each feature, which makes it possible to over-fit the feature selection criterion and you end up with a set of features that is optimal for this particular sample of data, but which gives poor generalisation performance. So not performing feature selection and using regularisation can often give better predictive performance.

I often use Bagging (form a committee of models trained on bootstraped samples from the training set) with ridge-regression models, which often gives an improvement in performance, and as all the models are linear you can combine them to form a single linear model, so there is no performance hit in operation.

  • $\begingroup$ Do you mean any $d-1$ points in a $d$ dimensional space? e.g. if you have 3 points in a 2-d space, with two being of class 1 and one being of class 2 all lying on a line, with the point of class 2 being in between the other two, then these cannot be separated with a line (1-d hyperplane) $\endgroup$ – tdc Dec 7 '11 at 16:58
  • $\begingroup$ It is normally assumed that the points are in "general position", so that (for example) they do not lie on a straight line, in which case in a 2-d space you can separate any 3 points. If all the points lie in a straight line then really they inhabit a 1-d subspace embedded in a 2-d space. $\endgroup$ – Dikran Marsupial Dec 7 '11 at 17:02
  • $\begingroup$ On wikipedia there is the statement "since the method averages several predictors, it is not useful for improving linear models" although I'm not sure why this should be true? $\endgroup$ – tdc Dec 7 '11 at 17:17
  • $\begingroup$ I don't see why that should be true either. I suspect the issue that a bagged linear model can be represented exactly by a single linear model, however the issue is the estimation of the parameters of the single model, not the form of the model. I have found bagging does improve generalisation, but the gain is generally small unless you have many more features than observations (so that the estimation of the model is unstable and a small change in the data produces a large change in the model). $\endgroup$ – Dikran Marsupial Dec 7 '11 at 17:30
  • $\begingroup$ Maybe you should update the Wikipedia page! You sound knowledgeable on the matter ... $\endgroup$ – tdc Dec 7 '11 at 17:52

Ridge regression, as the name suggests, is a method for regression rather than classification. Presumably you are using a threshold to turn it into a classifier. In any case, you are simply learning a linear classifier that is defined by a hyperplane. The reason it is working is because the task at hand is essentially linearly separable - i.e. a simple hyperplane is all that is needed to separate the classes. The "ridge" parameter allows it to work in cases that are not completely linearly separable or problems which are rank deficient (in which case the optimisation would be degenerate).

In this case, there is no reason why other classifiers shouldn't also perform well, assuming that they have been implemented correctly. For example, the SVM finds the "optimal separating hyperplane" (i.e. the hyperplane that maximises the margin, or gap, between the classes). The C parameter of the SVM is a capacity control parameter analogous to the ridge parameter, which allows for some misclassifications (outliers). Assuming the parameter selection process has been carried out diligently, I would expect the two methods to produce almost exactly the same results on such a dataset.

  • 2
    $\begingroup$ I remember reading it can be shown that LS-SVM binary classification is equivalent to Ridge Regression on -1,1 labels, their formulation are the same. $\endgroup$ – Firebug Jul 21 '16 at 17:23
  • $\begingroup$ Think you might be right on that one $\endgroup$ – tdc Jul 21 '16 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.