Assume I have a $2$-dimensional dataset $X=(x_1, x_2)$ where both features are not uniformly distributed over their respective ranges.

I now need to select $100$ datapoints from this dataset to be manually labeled, which will be used to train a 2-class SVM.

Should I choose these datapoints randomly or s.t. $x_1$ and $x_2$ are uniformly distributed?

Intuitively the latter makes more sense and should give a more accurate model, right? Unfortunately I can't find any source for that, even though I think it should be a common question. Probably just not googling right.


Well, if you can't assume randomness of data points in the set i.e. you assume the values are correlated with the order of the points in the set - you could make sample by uniformly taking the points (every n-th, where n = |P| / |s|; P is population, s is sample), but I would use the k-fold cross-validation technique (because I had the same dilemma as you and solved it by adopting this highly regarded validation technique):

The k-fold cross-validation provides fair accuracy estimation, using entire data set - when data insufficiency prevents separation of training and test data sets. The data is partitioned into k folds (exclusive, no data points shared among the folds), equal in size. Then, the validation is performed in k iterations, having different combination of k-1 training subsets and one test subset. Once all iterations are finished, the average values of the effectiveness measures (F1-score) are computed. Data points are distributed among folds randomly.

Btw. the Support Vector Machine is designed for binary (2-class) classification, it is not an option. [1] C. Cortes, V. Vapnik, Support-Vector Networks, Mach. Learn. 20 (1995) 273–297. doi:10.1023/A:1022627411411.

  • $\begingroup$ I don't think this answers my question. What I meant was that, if my dataset contains $x_1=1$ 3 times as often as $x_1=2$, should my sample contain $x_1=1$ as often as $x_1=2$ (uniformly sampled over the value range) or should I sample randomly (in which case my sample would also approximately contain $x_1=1$ 3 times as often as $x_1=2$). $\endgroup$ – Claas M. Sep 1 '18 at 17:03
  • $\begingroup$ From method standpoint: no, your sampling method should be agnostic on distribution of measured/collected values in the training dataset - because, you are proposing a method that should be generally applicable and not efficient only with this particular dataset. From technical/ML standpoint: the ML algorithms in general should learn quicker (i.e. become more precise, after less learning examples) if you provide more diverse training set. But, in case of SVM, I don't think you'll have that issue. Overall: think about k-fold cross-validation, it will boost cred. of reported results. $\endgroup$ – hardyVeles Sep 2 '18 at 2:51
  • $\begingroup$ Think about it like this: the point of your research is not to achieve good results but to propose a good method (algorithm/approach). Hence, your method shouldn't be designed to reflect that closely the features of dataset, being available during research. Sampling of the training subset is part of the method too, not something you do for yourself. (If) you can't know what will be features of input dataset, when someone else tries to use the method proposed by your research. $\endgroup$ – hardyVeles Sep 2 '18 at 2:57
  • $\begingroup$ (sorry, I forgot to give an answer :) ): in essence, if your dataset is collected by measuring independent variables: it doesn't meter if you take every n-th, first n-th or random n reads. If you ask me for simple answer: sample it randomly. If you ask me for what I would do: use the k-fold cross-validation as it will clear you from such worries and provide you realistic* insights on performances of your method.(btw. I recon k=5 for debug/dev phase, and k=10 for the reported validation results) * the F1-scores, measured with different datasets, will be within tighter range, if not ~the same $\endgroup$ – hardyVeles Sep 2 '18 at 3:04
  • $\begingroup$ Thanks, I like the argumentation that the point of my research is not to achieve good results but to propose a good method. $\endgroup$ – Claas M. Sep 3 '18 at 3:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.