I started using Radial Basis Function Networks for regression (see here for an overview of RBFNs). The specifics are:

  • $10^3 < n < 10^4$ input points, $x^{(i)} \in \mathbb{R}^d$, where $d$ is the input space dimensionality (usually $2 \le d \le 20$).
  • $n$ normalized squared exponential units, centered on the inputs (plus possibly a bias unit);
  • a single scale factor $\sigma$ for all units.

The number of inputs is low enough that I can find the weight matrix $W$ exactly by solving the linear system (e.g., backslash operator in MATLAB). However, I need to define a good scale factor $\sigma$, and RBFNs are notoriously prone to overfitting.

My current approach consists of performing leave-one-out cross-validation. The cross-validated loss is a jagged/noisy function of $\sigma$ due to cross-validation variability. So I am optimizing it via robust Bayesian Optimization (see here). This seems to work fine, but I wonder if I can do better -- there is a plentitude of methods for choosing hyperparameters and I am new to this specific field (used to work with Gaussian Processes).

What would you recommend as methods for picking hyperparameter(s) for this specific case?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.