Why atoms in the dictionary of Dictionary Learning method are not required to be orthogonal?

Question

According to Sparse Dictionary Learning (wiki),

Sparse dictionary learning is a representation learning method which aims at finding a sparse representation of the input data (also known as sparse coding) in the form of a linear combination of basic elements as well as those basic elements themselves. These elements are called atoms and they compose a dictionary. Atoms in the dictionary are not required to be orthogonal, and they may be an over-complete spanning set.

I think matrix factorization methods require their basis (a column of $W$ in $V \approx WH$) to be orthogonal? How does Dictionary Learning work differently from matrix factorization?

Answer 1

The part next to the part you highlighted,

they may be an over-complete spanning set

Will not be possible if the components are required to be orthogonal.

The reason for allowing overcompleteness is that if your dictionary has appropriate structure, you might be able to represent most interesting signals, or at least approximately represent them, using fewer nonzero entries (sparsity).

This is also elaborated in the further part of article you cite:

An overcomplete dictionary which allows for sparse representation of signal can be a famous transform matrix (wavelets transform, fourier transform) or it can be formulated so that its elements are changed in such a way that it sparsely represents the given signal in a best way. Learned dictionaries are capable of giving sparser solutions as compared to predefined transform matrices.

Basically this means that if your dictionary will behave reasonably, you might be able to represent data you are interested in using fewer nonzero numbers - this might not work as dimensionality reduction (because of overcompleteness), but still be useful, for example for data compression where only storage of nonzero components counts.