Explain log likelihood behaviour

Question

(This question is related to a previous one I made, here)

I have a set of 2D observations (measured data) of sample size $N$:

$$O = \{(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)\}$$

I also have a model $S(v_1, v_2, ..., v_p)$, which depends on $p$ parameters, and attempts to reproduce the observed data by generating artificial "observed" data.

The number of elements and spatial distribution of the artificial data generated with the model, depend on the values given to the $p$ parameters each time the model is used.

For example, here is some observed/measured data (left) of sample size $N=1000$, and some artificial data (right) of sample size $M=350$, obtained by fixing the $p$ parameters in $S$ to some values:

By varying the parameters of the model $S$, I generate $k$ sets of artificial data (each one uniquely described by the values assigned to the $p$ parameters) that I call $A_1, A_2, ..., A_k$, where:

$$A_r = \{(x_1, y_1), (x_2, y_2), ..., (x_{M_r}, y_{M_r})\}$$

(where the sets $A_1, A_2, ..., A_k$ have sample sizes $M_1, M_2, ..., M_k$)

I need to select from $A_1, A_2, ..., A_k$ the set that presents the best fit with my observed data $O$.

To do this I apply a log-likelihood test to each artificial data set:

$$L_r = -\sum_{i=1}^N \log ( \frac{1}{M_r} \sum_{j=1}^{M_r} e^{-\frac{(x_i - x_j)^2}{2}} e^{-\frac{(y_i - y_j)^2}{2}} )\;;\; r=1,..,k$$

where $(x_i, y_i)$ and $(x_j, y_j)$ represent observed and artificial data respectively.

As I understand it, I need to find the artificial set that minimizes the log-likelihood, and this will be the one that shows the best fit.

Something unusual (to me) happens with the likelihood values as the sizes of the artificial sets increase:

where each blue dot represents the log-likelihood value $L_r$ (y axis) obtained for an artificial set $A_r$ with sample size $M_r$ (x axis). The red line is the sample size for the observed data ($N=1000$).

Why is the log-likelihood small for artificial sets with few elements, climbs to a maximum and then starts dropping as the sample size of the artificial sets increase?

I would have expected the exact opposite behaviour, given that I am after the minimum log-likelihood value.

(The python code I used to generate these tests can be downloaded from here)

Answer 1

Several things could be wrong; I can only suggest some things to look at.

What's the smallest $N$ to debug with — 5, 10 ? Plot that with $N_r = N/2, N, 2N$ . Follow the steps, with rug plots or quantiles of $|p_i - q_j|, exp_{ij}$, row sums.
(It's not clear to me if $1 / M_r$ scaling is correct, maybe row sums 1 in the $N \times N_r$ array of $exp_{ij}$ ?)

$exp^{ - |p - q|^2 / r }$ is very sensitive to radius $r$. And it doesn't scale: as $N_r \to \infty$, the $exp$ s crowd near 1 (I believe). There are lots of other metrics between 2d point sets; average nearest-neighbor with nnear say 10 is fast and easy with scipy.spatial.cKDTree .

Can the data be viewed as a continuous terrain, 1 at the data points, falling off to 0 in between ? If so, the average error of the $N_r$ points interpolated at the $N$ sounds like a reasonable metric; see (ahem) inverse-distance-weighted-idw-interpolation-with-python. And terrains are nice to plot.