How important is basis expansion for deep nets? If deep neural nets are considered to be universal function approximators, is basis expansion really necessary? Or would this be case-specific? For example, if one has three quantitative X variables, would there be any advantage in expanding the number of variables by introducing interactions, polynomials, etc.? This seems to have good utility in e.g. RFs and SVM, but I'm unsure of whether this would be a good strategy for neural nets.
If this is perhaps too broad or vague, could someone point me to some pertinent information on basis expansion and feature engineering in the context of deep nets? 
 A: The idea of deep neural network is it can do the feature engineering automatically for us. (See the first chapter of the deep learning book.) I would strongly recommend you to read the first chapter. 
Doing basis expansion is not really necessary and uncommonly used. Keep in mind that, the deep net usually takes raw features as inputs, for images that have (at least) thousands of pixels, it is also not possible to do the basis expansion (say higher order polynomial expansion) effectively before feeding to the neural network. 

In fact, there are some operations in deep neural network can be viewed as basis expansion. 


*

*Convolution layer can be viewed as doing feature engineering in Fourier basis expansion. See my question: What is the intuition behind convolutional neural network? 

*The ReLU can be viewed as doing piecewise linear fit (spline basis). 
A: Many deep learning models learn their own features from the raw input data during training (e.g., 2D Convolutional Neural Networks for images). So in many cases, you don't even have to worry about passing variables explicitly to your model.
In some other cases, you still need features, but only core features (e.g., words in NLP). These features are represented as vectors in an embedding space that captures similarity (e.g., that 'president' is close to 'Obama'). The embedding space either comes from unsupervised pre-training (word2vec, glove) or is initialized randomly, and the vectors are tuned during training via backpropagation. The architecture of the network is responsible for learning feature combinations, like the difference between 'not bad, quite good' and 'not good, quite bad'.
The 'Feature combinations' paragraph of Section 3 of Goldberg, Y. (2015). A primer on neural network models for natural language processing. Journal of Artificial Intelligence Research, 57, 345-420. very well explains this (I really recommend reading the whole Section 3, it is excellent):

The combination features are crucial in linear models because they introduce more
  dimensions to the input, transforming it into a space where the data-points are closer to
  being linearly separable. On the other hand, the space of possible combinations is very
  large, and the feature designer has to spend a lot of time coming up with an effective
  set of feature combinations. One of the promises of the non-linear neural network models
  is that one needs to define only the core features. The non-linearity of the classifier, as
  defined by the network structure, is expected to take care of finding the indicative feature
  combinations, alleviating the need for feature combination engineering.

