You mention linear regression. This is related to logistic regression, which has a similar fast optimization algorithm. If you have bounds on the target values, such as with a classification problem, you can view logistic regression as a generalization of linear regression.
Neural networks are strictly more general than logistic regression on the original inputs, since that corresponds to a skip-layer network (with connections directly connecting the inputs with the outputs) with $0$ hidden nodes.
When you add features like $x^3$, this is similar to choosing weights to a few hidden nodes in a single hidden layer. There isn't exactly a $1-1$ correspondence, since to model a function like $x^3$ with sigmoids may take more than one hidden neuron. When you train a neural network, you let it find its own input-to-hidden hidden weights, which has the potential to be better. It may also take more time and it may be inconsistent. You can start with an approximation to logistic regression with extra features, and train the input-to-hidden weights slowly, and this should do better than logistic regression with extra features eventually. Depending on the problem, the training time may be negligible or prohibitive.
One intermediate strategy is to choose a large number of random nodes, similar to what happens when you initialize a neural network, and fix the input-to-hidden weights. The optimization over the *-to-output weights stays linear. This is called an extreme learning machine. It works at least as well as the original logistic regression.