1
$\begingroup$

I have 3 millions instances with 30 features each and I am trying to reduce it in a sensible size for my computer for a classification problem. What are possible methods I can use to reduce the data while keeping the quality of classification reasonable.

$\endgroup$
1
$\begingroup$

There are numerous systematic ways to approximate an empirical density (the data) both jointly and marginally and that may have advanatges over random sampling in that certain features (2 * n-1) of them can be preserved exactly.

If you recall the mean value theorem from calculus, there is a vaue x s.t. f(x) * (b - a) equals the integral exactly, this may seem less surprizing. Think of the constant function defined by f(x)*I(a,b) as a discrete distribution that approximates the continuous distribution f(x) and gets its mean exact. This extends to (2 * n-1) moments and to p dimensions and moments can be replaced by other features and then one redefines x on the circle and one ends up with good lattice points (matching trignometric moments) and one ends up in Fourier analysis and the FFT as the frontier applied statistics never got past.

OK but its the basis of quadrature, an easy start is called the von Mises step function approximation and quasi-random numbers are closely related. Perhaps google quantization.

Someone (Bill DuMouchel?) looked in to using these methods around 2001 to deal with massive data sets, but didn't realize how hard it was and I think gave up.

$\endgroup$
  • $\begingroup$ Could you elaborate on how this would help? $\endgroup$ – Placidia Nov 2 '12 at 15:58
  • $\begingroup$ If it is important to get say certain percentiles just right, you could match those of the population exactly in your sample. $\endgroup$ – phaneron Nov 5 '12 at 13:19
  • $\begingroup$ @Placidia: just came accross this link andrewgelman.com/2012/11/16554/#comment-108310 which provides a better elaboration (recall with 30 features a sampel size of hundreds is still very small due to curse of dimensionality) $\endgroup$ – phaneron Nov 7 '12 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.