Suppose I have a function $g_0\in L_2(\mathbb{R})$ such that we observe $(X_i,Y_i)$, $i=1,...,n$ such that

$$ \begin{align*} Y_i & = g_0(X_i) + V_i \end{align*} $$

We wish to estimate $g_0$, and since $g_0\in L_2(\mathbb{R})$, we can approximate $g_0$ by using an orthonormal series $\{h_k\}_{k\in\mathbb{N}}$. So for each sample size $n$, we approximate $g_0$ by considering the truncated series $g_{p_n}(x) = \sum_{k=1}^{p_n}\alpha_k h_k(x)$, such that $\alpha_k = \langle g,h_k\rangle$, and my estimator $\hat g = \sum_{k=1}^{p_n}\hat\alpha_kh_k$ will be defined as

$$ \begin{align*} \hat{\boldsymbol \alpha} = \arg\min_{\boldsymbol{\alpha}\in\mathbb{R^{p_n}}} \sum_{i=1}^{n}\left[Y_i - \sum_{k=1}^{p_n}\alpha_kh_k\right]^2\\ \end{align*} $$

The point is I wish to constrain the fitted coefficients $\hat{\boldsymbol\alpha}$ such that for each entry of $\hat{\boldsymbol \alpha}$, $\alpha_k$, I would like $|\alpha_k| \leq C$ for some $C$ strictly larger than $\|g_0\|_{\infty}$.

I understand that if we were to Heuristically argue this change, all we would do is threshold everything to be less than or equal to in magnitude to $C$, but how does this constraint change the standard "least squares" solution of $\hat{\boldsymbol \alpha}$ in closed form?


Your constraint does significantly alter the solution approach. In general, you obtain a quadratic programming problem, see here for example. It is quite more difficult to solve than the simple (ridge-regularized) least squares problem.

If it is really only one coefficient which is restricted, however, you can also use a different strategy: write your least squares functional as

$$ \begin{align*} L(\alpha_k; \boldsymbol \alpha_{-k})= \sum_{i=1}^{n} \left[\underbrace{\bigg(Y_i - \alpha_kh_k(X_i)\bigg)}_{=:Y_{-k,i}} - \sum_{\substack{j=1\\j\neq k}}^{p_n}\alpha_jh_j(X_i)\right]^2\\ \end{align*} $$

Here $\boldsymbol \alpha_{-k}$ denotes the $p_n-1$-dimensional vector $\boldsymbol \alpha$ without the $k$-th component $\alpha_k$, and $Y_{-k,i}$ is the new adjusted target. Now, in principle, you can do a one-dimensional minimization where you don't allow $\alpha_k$ to be outside your chosen boundary -- in practice: span a grid of points and search on it.

Such an approach might be easily extended to a larger number of constraint, but becomes quickly infeasible. If you have such extensions in mind, I would suggest to read the literature on quadratic programming (... which I will do later on because I also don't know the standard way to solve such a problem ... or better, my standard approach up to now was to take the program which does it for you).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.