1
$\begingroup$

I started using Radial Basis Function Networks for regression (see here for an overview of RBFNs). The specifics are:

  • $10^3 < n < 10^4$ input points, $x^{(i)} \in \mathbb{R}^d$, where $d$ is the input space dimensionality (usually $2 \le d \le 20$).
  • $n$ normalized squared exponential units, centered on the inputs (plus possibly a bias unit);
  • a single scale factor $\sigma$ for all units.

The number of inputs is low enough that I can find the weight matrix $W$ exactly by solving the linear system (e.g., backslash operator in MATLAB). However, I need to define a good scale factor $\sigma$, and RBFNs are notoriously prone to overfitting.

My current approach consists of performing leave-one-out cross-validation. The cross-validated loss is a jagged/noisy function of $\sigma$ due to cross-validation variability. So I am optimizing it via robust Bayesian Optimization (see here). This seems to work fine, but I wonder if I can do better -- there is a plentitude of methods for choosing hyperparameters and I am new to this specific field (used to work with Gaussian Processes).

What would you recommend as methods for picking hyperparameter(s) for this specific case?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.