(I'll answer the second question, about the results of your maximum likelihood with fixed differential entropy procedure; I don't know the answer to the first, about whether KDE can be phrased as optimizing some objective functional.)
Without additional constraints such as bounds on the derivatives, the problem is ill-posed: you can achieve any finite likelihood with fixed differential entropy.
Let $f_0$ be any density over $\Omega$ with differential entropy $h$. If $\log \lvert \Omega \rvert < h$, this is impossible and the problem is unsatisfiable; otherwise, take it to be e.g. uniform over some subset of $\Omega$ with appropriate volume.
Then define
$$
f(x)
= \begin{cases}
q & \text{if } x = x_i \text{ for some } i \\
f_0(x) & \text{otherwise}
\end{cases}
$$
for any finite $q$. This has likelihood $q^m$, where $m$ is the number of points $x_i$; since we've only modified $f_0$ on a set of measure zero, $f$ still has integral 1 and differential entropy $h$. So you can make the likelihood as high as you want.
Of course, changing the density function on a set of measure zero is in some ways "not fair"; the distribution itself won't be altered, for example.
But we can achieve essentially the same thing with continuous, or even $C^\infty$ density functions; just make a very narrow bump at each $x_i$. The next section shows formally that we can do that, but it's just this intuition in equations.
Let $f_0$ be a continuous (or $C^\infty$) density of differential entropy $Q$ whose support does not include any of the $x_i$ (or a ball around them). This makes the calculations much easier, as we don't have to worry about interactions between $f_0$ and the bumps we'll put at each $x_i$, but is not fundamental to the approach.
Let $\ell < \min_{i,j} \lvert x_i - x_j \rvert$ and also smaller than the distance from any $x_i$ to the support of $f_0$.
Define $$f_i(x) = \frac1\ell g\left( \frac{x - x_i}{\ell} \right),$$
where $g$ is the pdf of a mean-zero distribution with continuous (or $C^\infty$) density and support on a subset of the unit ball.
Many kernel functions
would work, for example.
Let $g(0) = g_0$ so that $f_i(x_i) = \frac1\ell g_0$,
and let $H[g] = \int g(x) \log g(x) \,\mathrm d x = \eta$
so that $H[f_i] = \eta - \log \ell$.
We'll take a mixture between $f_0$ and the $f_i$, for some $\alpha \in (0, 1)$ to be determined later:
$$
f(x) = (1-\alpha) f_0(x) + \frac{\alpha}{m} \sum_{i=1}^m f_i(x)
.$$
Then we have
$$
\int f(x) \,\mathrm d x
= (1 - \alpha) \int f_0(x) \,\mathrm d x
+ \frac{\alpha}{m} \sum_{i=1}^m \int f_i(x) \,\mathrm d x
= 1
,$$
and
$$L(f) = \left( \frac{\alpha}{m \ell} g_0 \right)^m = \left( \frac{g_0}{m} \right)^m \left( \frac{\alpha}{\ell} \right)^m
.$$
The first term is constant; we want to choose $\alpha$, $\ell$ so as to maximize $\alpha / \ell$ and hence $L(f)$.
Our constraint on $\alpha$ and $\ell$ is determined by
\begin{align}
\int f(x) \log f(x) \,\mathrm{d}x
&= \int (1-\alpha) f_0(x) \log\left( (1-\alpha) f_0(x) \right) \,\mathrm{d}x
+ \sum_{i=1}^m \int \frac{\alpha}{m} f_i(x) \log\left( \frac{\alpha}{m} f_i(x) \right) \mathrm d x
\\&= (1-\alpha) H[f_0] + (1-\alpha) \log(1 - \alpha)
+ \frac\alpha m \sum_{i=1}^m H[f_i]
+ \frac\alpha m \sum_{i=1}^m \log \frac\alpha m
\\&= (1-\alpha) Q + \alpha \eta - \alpha \log \ell + (1-\alpha) \log(1 - \alpha) + \alpha \log \alpha - \alpha \log m
.\end{align}
Setting this equal to $h$,
dividing both sides by $\alpha$,
and moving some terms,
we get
$$
\frac{Q - h}{\alpha} + \frac{1-\alpha}{\alpha} \log(1 - \alpha) + \log \frac\alpha\ell = Q - \eta + \log m
.$$
We could directly solve this for $\ell$, but the expression is little messy and we don't care about the particular value of $\ell$.
It's sufficient to note that as we drive $\ell \to 0$,
we'll have $\alpha \to 0$ as well; we care about the relative rate.
We can use
$$\log(1-\alpha) = - \alpha - \mathcal O(\alpha^2)$$
so that
$$
\frac{1-\alpha}{\alpha} \log(1-\alpha)
= - 1 + \alpha - \mathcal O(\alpha)
= -1 \pm \mathcal O(\alpha)
.$$
Thus
$$
\frac{Q - h}{\alpha} \pm \mathcal O(\alpha) + \log \frac\alpha\ell = 1 + Q + \log m - \eta
$$
and so
$$
\frac\alpha\ell
=
\underbrace{\exp\left( 1 + Q + \log m - \eta \right)}_\text{constant}
\exp\left( \frac{h - Q}{\alpha} \pm \mathcal{O}(\alpha) \right)
.$$
As long as $Q < h$,
we have $\frac\alpha\ell \to \infty$ as $\alpha \to 0$,
so $L(h) \to \infty$ as $\alpha \to 0$.
In the limit $\ell \to 0$, this converges to $f_0$ plus delta functions at each of the $x_i$ (as before). But for any finite $\ell$, we know we can balance the entropy of each nearly-a-delta-function spike with the entropy of $f_0$ appropriately so that the total is $h$. Thus, as before, for any finite target likelihood we can find a distribution with that likelihood and differential entropy $h$.
A constraint such as a Lipschitz density would correspond in this setting to a constraint on $\ell$, preventing us from driving the density to infinity.
It might be possible to derive the maximum-likelihood estimator among $L$-Lipschitz densities with differential entropy $h$. This might look something like:
- Put peaks as high as possible at each $x_i$.
- Decrease away from each peak uniformly with slope $L$.
- Stop decreasing at some base level uniform over $\Omega$.
This likely would not actually be the MLE in this family, but you might be able to get a linear program or something that gives a solution kind of like this.
