I have 3 millions instances with 30 features each and I am trying to reduce it in a sensible size for my computer for a classification problem. What are possible methods I can use to reduce the data while keeping the quality of classification reasonable.
There are numerous systematic ways to approximate an empirical density (the data) both jointly and marginally and that may have advanatges over random sampling in that certain features (2 * n-1) of them can be preserved exactly.
If you recall the mean value theorem from calculus, there is a vaue x s.t. f(x) * (b - a) equals the integral exactly, this may seem less surprizing. Think of the constant function defined by f(x)*I(a,b) as a discrete distribution that approximates the continuous distribution f(x) and gets its mean exact. This extends to (2 * n-1) moments and to p dimensions and moments can be replaced by other features and then one redefines x on the circle and one ends up with good lattice points (matching trignometric moments) and one ends up in Fourier analysis and the FFT as the frontier applied statistics never got past.
OK but its the basis of quadrature, an easy start is called the von Mises step function approximation and quasi-random numbers are closely related. Perhaps google quantization.
Someone (Bill DuMouchel?) looked in to using these methods around 2001 to deal with massive data sets, but didn't realize how hard it was and I think gave up.