Does a radial basis function network work in high dimensions?

Question

It seems that a single-layer radial basis function network with normalized weights is the same thing as kernel smoothing (see e.g. Haykin Neural Networks: a Comprehensive Foundation, Section 5.12).

It's well known that local smoothing methods don't work in high dimensions (say, more than ten dimensions) due to the curse of dimensionality. For example, $k$-nearest neighbors doesn't work very well at all (see e.g. Hastie, Tibshirani, and Friedman, Elements of Statistical Learning, section 2.5).

I have two questions:

1. I presume Nadaraya-Watson kernel smoothing suffers from the same problem as kNN. Is this the case? And the same for radial basis function networks: can they be used in high dimensions?

2. If these methods can't be used in high dimensions, then why do "kernelized" regression methods (like SVM or kernel ridge regression) work in high dimensions? In practice, people use them successfully on high dimensional inputs like images?

Answer 1

I don't know what Nadaraya-Watson smoothing is, but I can answer your second question.

Then why the "kernelized" regression (like SVM or kernel-ridge-regression) on the contrary will work in high dimension (as in practice people use them successfully on high dim inputs like images)?

The key difference is regularization. SVM and kernel ridge regression both impose constraints on model flexibility. The key point in the Elements of Statistical Learning discussion of kNN is that kNN and linear regression are viewed as two extremes of the bias-variance trade-off. Connecting this to SVM, note that the most common SVM with a regularization $C$ hyperparameter is use to constrain the model.