# What kind of kernel is used by statsmodels.nonparametric.kernel_regression.KernelReg?

I am doing multivariate nonparametric kernel regression using the Python function as mentioned in the title. The documentation can be found here: https://www.statsmodels.org/stable/generated/statsmodels.nonparametric.kernel_regression.KernelReg.html#statsmodels.nonparametric.kernel_regression.KernelReg

As far as my understanding goes, kernel regression requires to specify a kernel, such as Gaussian (sometimes called RBF). However, this function interface does not seem to allow me to specify the kernel other than 'll'(locally linear), 'lc'(locally constant), unless I am missing something. Can anybody explain to me what kind of kernel is used for 'll' or 'lc' specifications? Are there any other kernels I can choose from, where the bandwidth can be automatically chosen?

More generally, what python tool would you recommend for a nonparametric regression with multiple predictors, and relatively large number of samples (10000+)? With a mere 1000 samples and two predictors, the afore-mentioned tool already is taking quite some time. I have dozens of regressions to run, therefore I appreciate good speed and smoothness more than sophistication. My data are simulated and plenty. So my work is almost an noisy interpolation rather than regression. However, linear regression is definitely not going to work.

• I found sklearn's support vector regression (SVR) to be much faster than statemodels' kernel regression. I thought they are very similar things. Why is SVR so much faster, taking seconds for synthesised data, while the latter takes minutes? I thought both algorithms need to evaluate the kernel N^2 times, where N is the number of (vector) samples? Maybe SM uses more sophisticated cross validation, while SVR simply uses default value? Which would you recommend for plenty of data? – user138668 May 17 '19 at 2:33

The two models that you mention in your comment are not at all the same thing:

statsmodels.nonparametric.kernel_regression.KernelReg I think refers to Nadaraya-Watson kernel regression, where $$\hat y(x) = \frac{\sum_{i=1}^n k(X_i, x) y_i}{\sum_{i=1}^n k(X_i, x)}.$$

Exactly what $$k$$ they use is indeed underdocumented in statsmodels. The source has some references for _est_loc_linear that I haven't looked at, but they presumably establish what the locally linear method is; I also haven't dug through the code to understand what it's doing. _est_loc_constant doesn't have any references. It seems that they're using econometrics terminology I'm not familiar with; presumably this is all clear to the person who wrote it, but I don't know.

scikit-learn's SVR is a totally different model, based on a regression-based support vector machine using the kernel trick. They're really not very comparable.

Part of the reason SVR is so much faster is that it uses the optimized LIBSVM library, whereas statsmodels is (I think) all written in not-particularly-optimized Python. There are also some algorithmic differences; SVR can throw away the non-"support vectors," while statsmodels's KernReg is probably computing all pairwise differences. But, also, SVR is not doing any bandwidth tuning unless you explicitly tell it to (with some technique in sklearn.model_selection), whereas I think KernelReg does bandwidth selection automatically.

I recently needed to do kernel estimation as well and stumbled upon the KernelReg function in the statsmodels library. And I couldn't seem to find any documentation either beside, as you mention, the choice of a local constant or local linear regression fit. I did a deep dive into the source code of the kernel_regression module. The following code is copied from the source and is where the weighting (using the kernel of interest) of the data happens

ker = gpke(bw, data=exog, data_predict=data_predict,
var_type=self.var_type,
#ukertype='aitchison_aitken_reg',
#okertype='wangryzin_reg',
tosum=False) / float(nobs)


So I looked for the function gpke in the module _kernel_base and found the following definition:

, data_predict, var_type, ckertype='gaussian',
okertype='wangryzin', ukertype='aitchisonaitken', tosum=True):


with the explanation of ckertype

"""ckertype : str, optional
The kernel used for the continuous variables."""


The variable ckertype is then later looked up in a dictionary that contains the corresponding kernel class.

## Conclusion

It seems that by default - in the continuous case - the kernel used as the weights in the regression is the Gaussian kernel.