2
$\begingroup$

Summary (details below!)

I have a basis expansion of $m$ (univariate) Gaussian PDFs to model the density of a sample $X$. The means of these PDFs are spaced equidistantly through the domain of $X$ and their bandwidth/std is equal for all, called $h$.
Given this scenario, is there a way to determine $h$ given $M$ and the range of $X$ so that $X$ is 'covered sufficiently' (as in, given the right basis weights, the samples density is approximated well by using this $h$)?


Setting

Consider a basis expansion $f(x)$ of $m$ bases that is used to model the density of a univariate sample $X$. For this the basis functions are chosen to be Gaussian density functions $\mathcal{N}(x|\mu_i,h)$ with their means $\mu_1,...,\mu_m$ being equidistantly spaced throughout the range of $X$. The basis expansion then is defined as

$$f(x)=\sum_{i=1}^{m} \mathcal{N}(x|\mu_i,h) \: w_i$$

with $w_i$ being the basis weight/coefficient of the i-th basis. These weights can be determined via MLE.

For visualization, figure 1 shows the basis functions spaced across the sample space, while figure 2 shows the weighted basis functions as well as the density as given by $f$.

The described model therefore has 2 hyperparameters: the number of basis function $m$ and the bandwidth of the bases $h$. So far I simply determined these via a grid search, seeing which combination of parameters best described a held out sample.

Question

Since the choices for $m$ and $h$ are dependent on each other in so far as a smaller $m$ requires a larger $h$ and a larger $m$ requires in turn a smaller $h$, I was wondering if the model selection could be simplified by determining $h$ merely depending on $m$ and the range of the data?

For this I did some experimentation: I defined the 'range' of $X$ to be covered as the difference between the 99th and the first percentile (noted as IPR) and formulated a simple equation for the bandwidth given as

$$h_{scale} = \frac{IPR}{m} \cdot \mathrm{scale} $$

I fitted some models with 9 values for $m$ and 2 values for $scale$, 0.25 and 0.5. When comparing them to the error contours I got from the sensitivity analysis experiments I conducted for different purposes, I was positively surprised. The results can be seen in fig. 3 and 4 on two different data sets. Note that the conditions of these later experiments and the ones during the prior model selection were identical.

In both plots the contour colors show the MAE of the models density estimates to the true densities of the respective data set. Figure 3 shows that a scale value of 0.25 works well, while in figure 4 0.5 seems the better choice.

Maybe my formulation for $h$ was wrong, but I assume a reasonable approximation can be done.
How could a fitting $h$ be determined in this scenario? To me it seems reasonable using $m$ and the range of $X$ to determine $h$, but I don't know how these could be better combined.

Figures

Figure 1 Five gaussian PDFs spaced out over the domain of a sample X

Figure 2 The five PDFs from figure 1, but with weights applied to them and their summed density being plotted as 'combined'.

Figure 3

Figure 4 enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.