What is low rank linear layer in neural networks? Going through the paper Convolutional Neural Network for Small-footprint Keyword Spotting. In the paper, authors have used low rank linear layer after convolution and max-pool layer.
What is the purpose of using this layer and how it is different from dense layer?
 A: The linked paper is not explicit on what they mean, however the most natural interpretation is the one suggested by @shimao, i.e. with "linear low-rank layer" they refer to a linear layer whose weight matrix is constrained to be of low rank.
In practice, if the transformation goes from $\mathbb{R}^n$ to $\mathbb{R}^m$ this can be implemented by:
$$ f(x) = UVx + b, \quad U \in \mathbb{R}^{m\times k}, V \in \mathbb{R}^{k\times n}, $$
with $k < min(n, m)$ being the desired rank for $UV \in \mathbb{R}^{m\times n}$.
As you can see, the number of parameters for this linear layer is now $mk+nk$ instead of $mn$ (ignoring the bias), which for small values of $k$ means a lower number of parameters. This contributes to the regularization of the model and to a higher time efficiency for training and evaluation.
A: It's not very explicit, but it just seems to be a linear transformation $ T(x) = Ax + b$ where $A,b$ are learned. In other words, it is just a dense layer without the non-linearity. It's not clear why they call it "low-rank"; perhaps they were thinking more along the lines of low VC dimension or something, compared to a regular fully-connected layer.
