Forcing smoothness of regression coefficients I'm building regression models on spectral datasets: the predictors are the intensites of signal at the different frequencies. In this case the intensities at close frequency values are highly correlated, being part of the same signal peak. I was consequently expecting regression coefficients varying slowly with the frequency, but this is not the case for several regression methods including PLS, Ridge and Lasso.
I looked for regression methods that somehow force the coefficients to be smooth function of the frequency, but without success, so I'm asking if there is any.
In order not to make the question too vague I try to ask two specific questions:


*

*Is there a regression method that uses a kind of penality, similar to that of the Ridge or Lasso regression, but involving the discrete derivatives of the regression coefficients instead of their values?

*Is there a regression method that uses regression coefficients that are smooth function of the frequency (polynomial, spline, whatever)?


Please point to references.
 A: If you want something simple, I would look at (weighted) fused LASSO or similarly ridge-style regression. Suppose you have
$$ y \in \mathbf{R}^{T\times 1} \text{ response} \qquad X_{k} \in \mathbf{R}^{T\times 1} \text{ where } k\in\{1, \dots, K\} \text{ $k$th covariate }$$
You want some smoothness of coefficients that should be close, ie if $k$th and $j$th covariates should have coefficients that are close in some norm. For simplicity, assume chain-type structure, ie $1$st covariate is related to $2$nd, $2$nd to $3$rd ect. The estimators you may want to look at are
$$ \hat\beta \in \underset{\beta\in \mathbf{R}^{K}}{ \text{arg min}}\|y-X\beta\|^2_T + 2\lambda_1 \|\beta\|_1 + 2\lambda_2 \|D \beta\|_1 \quad \text{fused LASSO}$$
or
$$ \hat\beta \in \underset{\beta\in \mathbf{R}^{K}}{ \text{arg min}}\|y-X\beta\|^2_T + 2\lambda \|D \beta\|^2_2 \quad \text{Ridge-type}$$
where
$$
D  = \begin{pmatrix}
0 &  0 & 0 & \dots & 0 & 0 \\
1 & -1 & 0 & \dots & 0 & 0  \\
0 & 1 & -1 & \dots & 0 & 0  \\
\vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\
0 & 0 & 0 & \dots & 1 & -1  \\
\end{pmatrix}.
$$
and $\|u\|^2_T = \langle u,u \rangle/T$. You may also add LASSO-type constraint to the second estimator to impose sparsity on top of $\ell_2$ constraint (you will get something like the elastic net). As usual, Ridge (without additional LASSO constraint) has closed form (provided that $D$ satisfies some simple conditions).
The first reference for the fused LASSO I would look at is: https://web.stanford.edu/group/SOL/papers/fused-lasso-JRSSB.pdf. There are many efficient solvers in many software packages for such type of problems.
Lastly, you may use correlation as weights for $k$ and $j$ pair of covariates and adjust the penalty term accordingly. Define $\rho_{k,j}$ as a simple correlation between two covariates. You probably want to have a correlation between two consecutive covariates, i.e. $\rho_{k,k+1}$, $k \in \{1,\dots, K-1\}$, and you want to take the absolute value of correlations. Then, redefine the $D$ matrix as
$$
D  = \begin{pmatrix}
0 &  0 & 0 & \dots & 0 & 0 \\
|\rho_{1,2}| & -|\rho_{2,1}| & 0 & \dots & 0 & 0 & 0 \\
0 & |\rho_{2,3}| & -|\rho_{3,2}| & \dots & 0 & 0 & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots \\
0 & 0 & 0 & \dots & 0 & |\rho_{K-1,K}| & -|\rho_{K,K-1}|  \\
\end{pmatrix}.
$$
where $\rho_{k,k-1} = \rho_{k-1,k}$, of course. Then you take into account the relative strength between each pair of covariates, which can help.
You have to try several estimators and see what works best for your dataset.
For your second question, maybe you can use the dictionary of functions, say $W$, to approximate the weight function of different frequencies you have in your dataset. So, if your predictors are $Z_{j,k}$, for $j\in \{1,\dots,J\}$, define $X_{k}=(Z_{1,k},\dots,Z_{J,k})W$, and run the usual regression of $y$ on $X$. $W$ can be splines for example. I'm not sure if it makes sense to smooth coefficients in this case.
Hope this helps.
A: This is the motivation for  Functional Data Analysis.
I will describe the functional covariate and scalar response linear model which is just one of many many possible models.
Asumes pairs $(x, y)$ where $$ y = \int_{a}^{b} \beta(t)x(t) \, dt + \varepsilon$$
Here $x$ is a random curve, y a scalar response and $\beta$ an unknown function that links both variables.
There are many approachs to estimate $\beta$ but one that is a direct solution to your problem is finding $\alpha$ in a suitable finite dimentional space such that minimices
$$ \sum_{i=1}^n \left(y_i - \int_a^b \alpha(t) x_i(t) \,dt\right)^2 + \lambda \| \alpha^{(m)} \|^2 $$
Usually $m=2$ so you penalize roughness of your function. Higher values of $\lambda$ will give smoother estimations.
Usually one does not observe curves but a discretized version of them.  If $(t_k, x_i (t_k))$ are your values for frecuency an intensity $1\leq k\leq p$ and $1 \leq i \leq n$ then you may estimates your curves with splines or any other method of your choice. A great plus of this method is that you do not need to have all your observations in the same grid. You can actually have different $p_i$. An other important aspect to consider is that these method ares specially designed to bypass the high correlation between the $x_i(t_k)$ $x_i(t_{k+s})$. Also using a suitable finite dimentional space make optimization tasks simple.
A great book for studing this models is Functional Data Analysis by Ramsay And Silverman.. 
There are also R packages such as fda and fda.usc.
