Asymptotic normality with weighted sum of objective function $\min_{x} \; f_n(x) + g_n(x)$

Question

Suppose

1. $f_n(x)$, $g_n(x)$ are convex functions w.r.t. $x$
2. the optimal point of the two problems $\min_x f_n(x)$ and $\min_x g_n(x)$ have asymptotic normality as $n \rightarrow \infty$
3. they converge to the same point $x_0$

$$ x_1 \overset{p}{\sim} N(x_0,\frac{1}{n}D_1), \quad where \quad x_1 = \arg\min_x f_n(x), $$ $$ x_2 \overset{p}{\sim} N(x_0,\frac{1}{n}D_2), \quad where \quad x_2 = \arg\min_x g_n(x). $$

Question: Do we still have asymptotic normality for the following optimization problem $$ \min_x \; f_n(x) + g_n(x) \;? $$

---

Here is an example on how the asymptotic normality and $n$ appear.

Consider data pairs $(u_i,v_i)$, $i=1,\cdots,n$ randomly generated from a linear model $$ u = k \cdot v + \epsilon , \quad where \quad \epsilon \sim N(0,1) . $$ Note that the variance is fixed.

We can do least squares regression and obtain an estimator $k_1$. As $n \rightarrow \infty$, there is asymptotic normality.

We can also do least absolute deviation regression and obtain an estimator $k_2$. As $n \rightarrow \infty$, there is asymptotic normality.

$k_1$ and $k_2$ converges to the same true value $k$ as $n \rightarrow \infty$.

Consider the following problem $$ \min_k \; \sum_{i=1}^n (u_i - kv_i)^2 + \sum_{i=1}^n |u_i - kv_i|. $$

Answer 1

Ok, so your example has two convex functions. I'm going to divide each of them by $n$ so they are averages of $n$ random convex functions. I'm also going to write the argument as $\theta$, since it's a parameter to be estimated. I'm also going to assume iid data $(X,Y)$, which is stronger than necessary, but convenient.

Convexity implies that $h_n(\theta)=f_n(\theta)+g_n(\theta)$ is convex. Also, since it is convex, the pointwise convergence (in probability) of $h_n(\theta)$ to a limit $h(\theta)$, which follows from a law of large numbers, is uniform on compact sets. This is enough to ensure that the minimiser $\hat\theta_n$ converges in probability to the minimiser $\theta_0$ of $h(\theta)$. (Well, together with existence of some moments, which I'm usually happy to assume)

Asymptotic normality requires a bit of smoothness. The classical condition is on the third partial derivatives of the objective function, but that's too strong for LAD regression. Write $h^*(\theta;x,y)$ for the function such that $h_n(\theta)=\frac{1}{n}\sum_i h^*(\theta; x_i,y_i)$ -- the single-observation objective function. Assuming independence of observations, it's enough (van der Vaart Asymptotic Statistics Theorem 5.23) that $h^*(\theta;X,Y)$ is differentiable at $\theta_0$ for almost every $(X,Y)$, that $h^*(\theta)$ is Lipschitz on a neighbourhood of $\theta_0$ (automatic for convex functions) and that the expected value $E_{X,Y}[h^*(\theta; X,Y)]$ has a second-order Taylor expansion at $\theta_0$. This does cover LAD regression (under some assumptions on $X$ and $Y$)

Now, if $f^*$ and $g^*$ (defined analogously to $h^*$) satisfy the smoothness conditions, then $h^*$ also will satisfy them, and $\hat\theta_n$ will be asymptotically Normal.

I don't know if it's automatically sufficient that $f^*$ and $g^*$ are consistent and asymptotically Normal (I suspect it would be possible to construct some pathological counterexample), but if you have a proof that the minimisers of $f_n$ and $g_n$ are asymptotically Normal, you probably have a proof of sufficient smoothness that translates to $h_n$ and thus a proof that the minimiser of $h_n$ is asymptotically Normal. And in particular it's true for LS and LAD regression.

There are, however, a couple of complications with this as an estimation technique. First, there's no guarantee that $f_n$ and $g_n$ are on remotely comparable scales, so a simple sum may well be a very suboptimal way to combine them. Second, the combined function $h_n$ will have the same roughness as the rougher of the two functions near $\theta_0$ and the same long-range growth as the faster-growing of the two functions far from $\theta_0$. This is typically the opposite of what you want: eg, the Huber $M$-estimator is set up so its influence function looks like the mean when $\theta$ is near $\theta_0$ and like the median when $\theta$ is far from $\theta_0$.