In This Stanford Tutorial, it says

"Sparse coding is a class of unsupervised methods for learning sets of over-complete bases to represent data efficiently...

While techniques such as Principal Component Analysis allow us to learn a complete set of basis vectors efficiently, we wish to learn an over-complete set of basis vectors...

The advantage of having an over-complete basis is that our basis vectors are better able to capture structures and patterns inherent in the input data."

Is there a good reason why over-complete bases are better able to capture structures and patterns inherent in the input data?

People say sparsity is desirable, but if sparsity requires increasing the dimensionality of your data, how do you decide whether you care about sparsity or the curse of dimensionality more?


1 Answer 1


Since nobody has attempted a response to this, I thought I'd give it my best shot. I feel like this is a partial answer at best, but here goes!

Neuroscience people like sparsity because it seems to be how our brain works. Other people like sparsity in neural networks for the same reason why SVM people use kernels. The "kernel trick" in SVMs is where you implicitly embed your points in a higher-dimensional space so that the classes which were previously not separable by a linear subspace become linearly separable. There's a higher chance that the function you're trying to learn is a "simple" one when thought of as a function of more features.

Of course, I have no proof of this and I'd love to see a better answer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.