I’m working on a classification problem. However, my training dataset is very small (just 800 items in training dataset) and each data item contains a small number of features (just 5 features). Firstly, I used Logistic Regression to create a model for this dataset. Unfortunately, prediction accuracy of my model was very bad. Next, I used Neural Network model, but could not see any progress.

I suspect, number of training data items and number of features in each item are not enough for training Logistic Regressing and Neural Network.

So my question is what are the good techniques for modeling small datasets?

  • 3
    $\begingroup$ What are your results? Are you sure that you can classify your dataset? 800 items is not that small and you should be able to do it. $\endgroup$
    – ThiS
    Jan 8, 2013 at 8:28
  • 1
    $\begingroup$ I'm getting roughly 76.00 +/- 4.00; I also think dataset with 800 items is a reasonable dataset. However, I think 5 features is not enough for getting a good result. $\endgroup$
    – Upul
    Jan 8, 2013 at 8:36

3 Answers 3


Neural networks can be notoriously difficult to work with (too many potential difficulties, such as local minima, over-fitting etc.). A kernel method is likely to be easier to work with, such as kernel ridge regression, the support vector machine, kernel logistic regression or Gaussian process classifiers, using a radial basis function kernel/covariance function. As long as the hyper-parameters are tuned using a sensible procedure (e.g. cross-validation), they are likely to provide good results with much less effort/risk than neural networks.


It might be a data issue and not a classifier issue (Neural Networks, Logistic Regression ..etc) problem, because when data has a shape ( say it could be divided into two classes ) 800 points should be fine as training set.

It might be that the decision boundary is a funny one (Not linear for example), have you tried to play with the Neural Network classifier parameters such as the number of hidden layer ..etc?

The problem with Neural networks is that the objective function is not always a convex function which translates into a bad parametrization of the classification problem. Have you tried kernel methods ( this is supposed to do a transformation on your data, in some cases this allows you to have an infinite number of feature if you use a suitable kernel ), you can use SVM for example, this is how it works:

#  svm
Svm = svm(Species ~ ., data = train,type = "C-classification",kernel = "polynomial",degree = 1, gamma = 2, cost = 0.5, coef0 = 2)
SvmPr = predict(Svm,test)

And, once again, you should know what the classifier parameter are meant to do and have a play around with them.



  • 4
    $\begingroup$ The convexity of SVMs is IMHO a rather overstate benefit. Once you include tuning the hyper-parameters (which is the key to their correct use), overall the problem is no longer convex. The key benefit of kernel methods is simply that they encourage the user to deal with over-fitting by carefully tuning the regularisation parameter $C$. $\endgroup$ Jan 8, 2013 at 9:45
  • $\begingroup$ @DikranMarsupial, I agree with you! people always say svm is very good for small dataset (even just 10 data points), I just don't get it why and what makes svm so special for such small dataset. If it's the overfitting issue here, well, yes, C param in svm could be tuned to counter the overfitting effect, but why wouldn't logistic regression do the job? LR also have regularization params to tune. $\endgroup$
    – avocado
    Oct 19, 2017 at 3:52
  • 1
    $\begingroup$ @avocado yes, IIRC there are similar generalisation bounds for regularised logistic regression, it is largely the regularisation that explains the good performance of SVMs rather than the hinge loss (IMHO). $\endgroup$ Oct 23, 2017 at 8:00

Given how small the data set is, have you tried investigating it using any non-parametric methods? For example, I bet you could easily do a full leave-one-out cross validation test using a nearest neighbour classifier on your 800 sample training set, and see what the estimated classification error rate is then. If it's low then there's almost certainly some way you can improve performance further. If it's high then things look less good...

  • $\begingroup$ Agreed. Another procedure would be to make visualizations to investigate how the data is separated -- this usually helps in choosing better kernels or ML methods. $\endgroup$ Mar 13, 2015 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.