1
$\begingroup$

I would like to generate some artificial data to evaluate an algorithm for classification (the algorithm induces a model that predicts posterior probabilities).

These are some basic properties of the dataset:

  • Features have to be continuous
  • Response variable is dichotomous (either 0 or 1)

I would like to test whether the algorithm can cope with:

  • Many feature / high dimensional problems
  • noise (it can drop features)
  • Multi-modality
  • ??? (how do I simulate correlation etc.)

I intend to implement the algorithm in R or Matlab. I can sample from multivariate normal distributions and specify a covariance matrix.

I would appreciate any feedback.

$\endgroup$
4
  • $\begingroup$ Have you looked into using a real-world pre-existing dataset? There are many advantages to using real-world data instead of synthetic data. I can immediately think of at least one type of problem that follows your criteria: vision classification. For example, in the MNIST dataset the features are pixel values and the response variable is a handwritten digit ('0'-'9'). edit: Just saw that you want binary classification, in the case of MNIST you could subset the data, for example try to classify zeros vs non-zeros. $\endgroup$
    – rm999
    Jan 23, 2012 at 15:05
  • $\begingroup$ Sorry I disaggree. As I produce probabilities an artifical generator would also allow me to test the calibration. This would (IMHO) be impossible using real-world datasets. It would also be hard to see under which conditions (e.g. above 10 higly correlated features) the algorithm starts to fail ... etc. $\endgroup$
    – cs0815
    Jan 23, 2012 at 15:20
  • 1
    $\begingroup$ Please, keep us informed on your progress with your related question on SO, stackoverflow.com/q/8972244/420055. $\endgroup$
    – chl
    Jan 23, 2012 at 18:02
  • $\begingroup$ I will have a look at the R mlbench package today. $\endgroup$
    – cs0815
    Jan 24, 2012 at 8:35

1 Answer 1

1
$\begingroup$

Some idea might be to generate something like the Madelon set from NIPS 2003 challenge; it fits your requirements pretty well.

You can generate a set like this starting with mlbench.xor (or mlbench.hypercube, might be easier) form mlbench package, then you combine classes it generated into two groups to make the dichotomous response and add new attributes to increase dimensionality -- some being random linear combinations of the original ones, some being just random noise.

$\endgroup$
1
  • $\begingroup$ MADELON has train.labels only, no test.labels not valid.labels. (In fact all the NIPS 2003 datasets on uciml, arcene dexter dorothea gisette, have only train.labels. Can anyone find the test.labels ?) $\endgroup$
    – denis
    Feb 5, 2012 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.