What are good techniques for modeling small datasets?

Question

I’m working on a classification problem. However, my training dataset is very small (just 800 items in training dataset) and each data item contains a small number of features (just 5 features). Firstly, I used Logistic Regression to create a model for this dataset. Unfortunately, prediction accuracy of my model was very bad. Next, I used Neural Network model, but could not see any progress.

I suspect, number of training data items and number of features in each item are not enough for training Logistic Regressing and Neural Network.

So my question is what are the good techniques for modeling small datasets?

Answer 1

Neural networks can be notoriously difficult to work with (too many potential difficulties, such as local minima, over-fitting etc.). A kernel method is likely to be easier to work with, such as kernel ridge regression, the support vector machine, kernel logistic regression or Gaussian process classifiers, using a radial basis function kernel/covariance function. As long as the hyper-parameters are tuned using a sensible procedure (e.g. cross-validation), they are likely to provide good results with much less effort/risk than neural networks.

Answer 2

It might be a data issue and not a classifier issue (Neural Networks, Logistic Regression ..etc) problem, because when data has a shape ( say it could be divided into two classes ) 800 points should be fine as training set.

It might be that the decision boundary is a funny one (Not linear for example), have you tried to play with the Neural Network classifier parameters such as the number of hidden layer ..etc?

The problem with Neural networks is that the objective function is not always a convex function which translates into a bad parametrization of the classification problem. Have you tried kernel methods ( this is supposed to do a transformation on your data, in some cases this allows you to have an infinite number of feature if you use a suitable kernel ), you can use SVM for example, this is how it works:

######################################
#  svm
######################################                           
Svm = svm(Species ~ ., data = train,type = "C-classification",kernel = "polynomial",degree = 1, gamma = 2, cost = 0.5, coef0 = 2)
SvmPr = predict(Svm,test)
table(as.vector(SvmPr),iris[-tr,5])

And, once again, you should know what the classifier parameter are meant to do and have a play around with them.

HTH

D

Answer 3

Given how small the data set is, have you tried investigating it using any non-parametric methods? For example, I bet you could easily do a full leave-one-out cross validation test using a nearest neighbour classifier on your 800 sample training set, and see what the estimated classification error rate is then. If it's low then there's almost certainly some way you can improve performance further. If it's high then things look less good...