Poor regression results of neural networks on 2d benchmark data (compared to spline interpolation)

Question

I try to understand for which regression tasks neural networks might be useful. One benchmark for me is to reproduce the ability of scipy.interpolate.griddata: Regression of the 2d function

$$ x(1-x)\cos(4\pi x) \sin^2(4\pi y^2). $$

However, this seems to be hard for neural networks:

Source code: https://nbviewer.jupyter.org/github/cknoll/demo-material/blob/master/compare_regression_mlp_vs_spline.ipynb

Questions:

1. Am I doing something wrong, e.g. in terms of meta parameters?
2. Is this task not suited for neural networks?
3. Would it be better to try TensorFlow or pytorch instead of scikit.learn?

Answer 1

In general a MLP should be capable for your task, but when using neural nets there are several things one should bear in mind:

• there are no strict rules for adjusting the hyperparameters and the whole network structure (deep and width) because it is strongly problem dependent

• therefore, fiddeling with the hyperparameters is important (you could use search algorithm instead of guessing)

• due to random initialization of the network's weights, it is worth to start the training multiple times to avoid unfavorable initializations

I could achieve much better results with only a few manual tries by using the deep learning library Keras:

The main problem with the implementation of scikit-learn's MLP is the tol parameter, which prevents the solvers sgd and adam from training further. Similar, the solver lbfgs stops quite early. As one can see from the Keras loss plot, there is a plateau before the 'main learning' begins:

Furthermore you should be aware of the difference between interpolation and regression, because they address different problems. Thus SciPy griddata won't be a good choice for noisy data: How is interpolation related to the concept of regression?

My modified version of the notebook.