I have 3 millions instances with 30 features each and I am trying to reduce it in a sensible size for my computer for a classification problem. What are possible methods I can use to reduce the data while keeping the quality of classification reasonable.
$\begingroup$
$\endgroup$
1
-
2$\begingroup$ What's wrong with random sampling? $\endgroup$– Peter FlomCommented Nov 2, 2012 at 11:05
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
There are numerous systematic ways to approximate an empirical density (the data) both jointly and marginally and that may have advanatges over random sampling in that certain features (2 * n-1) of them can be preserved exactly.
If you recall the mean value theorem from calculus, there is a vaue x s.t. f(x) * (b - a) equals the integral exactly, this may seem less surprizing. Think of the constant function defined by f(x)*I(a,b) as a discrete distribution that approximates the continuous distribution f(x) and gets its mean exact. This extends to (2 * n-1) moments and to p dimensions and moments can be replaced by other features and then one redefines x on the circle and one ends up with good lattice points (matching trignometric moments) and one ends up in Fourier analysis and the FFT as the frontier applied statistics never got past.
OK but its the basis of quadrature, an easy start is called the von Mises step function approximation and quasi-random numbers are closely related. Perhaps google quantization.
Someone (Bill DuMouchel?) looked in to using these methods around 2001 to deal with massive data sets, but didn't realize how hard it was and I think gave up.
-
$\begingroup$ Could you elaborate on how this would help? $\endgroup$– PlacidiaCommented Nov 2, 2012 at 15:58
-
$\begingroup$ If it is important to get say certain percentiles just right, you could match those of the population exactly in your sample. $\endgroup$– phaneronCommented Nov 5, 2012 at 13:19
-
$\begingroup$ @Placidia: just came accross this link andrewgelman.com/2012/11/16554/#comment-108310 which provides a better elaboration (recall with 30 features a sampel size of hundreds is still very small due to curse of dimensionality) $\endgroup$– phaneronCommented Nov 7, 2012 at 13:38