Estimating hyperparameter in basis functions (Gaussian and sigmoid) for linear regression

Question

I am working on a linear regression problem with Gaussian and sigmoid basis functions. My data set is very large, say a total of 15K inputs with each input having 46 features. I have divided my data into three sets as usual Training, CV and Test. By plotting the ERMS vs Degree of Polynomial for all three sets I chose the degree. But I am clueless about how to choose the parameters ($S$ and $\mu$) in basis functions. I am using the following: $$ F_{\text{Gaussian Basis}}(x[j])= \exp\left(\frac{(x-\mu(j))^2}{S^2}\right). $$ For now I am choosing them randomly. Any suggestion or good link where I can read about this?

Answer 1

The best thing to do is probably to choose the hyper-parameters to minimise the error on the validation set. This is quite easy to do using standard numerical optimisation tools (I like the Nelder-Mead simplex algorithm as it doesn't reuire gradient information). However, for non-linear models it is probably best to use ridge regression, rather than ordinary least squares regression in order to avoid over-fitting. The ridge parameter can be optimised in the same way as the hyper-parameters of the basis functions. As the hyper-parameters need to be strictly positive, optimise the logarithms of the hyper-parameters to get a convenient uncontrained optimisation problem.