-1
$\begingroup$

Suppose I have a linear model:

$$ y_i = \sum_{j=1}^{p} \beta_{j} x_{ij} $$

Having $p$ variables, and $n$ samples. Moreover, predictor variables considered can be divided into two groups, where variables from a group are uniquely associated to a variable from the second group and vice versa. Calling the two set of variables $p_1$ and $p_2$, they will have same length. Changing the model to: $$ y_i = \sum_{j=1}^{p/2} \gamma_{j} \big (\beta_{j}^{p_1}x_{ij}^{p_1} + \beta_{j}^{p_2}x_{ij}^{p_2} \big ) $$ Where $x_{j}^{p_1}$ is data of the $j^{th}$ variable in sample $i$ from variables set $p_1$. $\beta_{j}^{p_1}$ is its corresponding model coefficient, while $\gamma_{j}$ is an overall multiplicative parameter.

Considering that I'm interested in the exploration of the estimated parameters, [question 1] is it useless to introduce such "multiplicative parameter" $\gamma_{j}$ inside the model?

By minimizing MSE and adding regularizers for parameters, [question 2] is it still possible to obtain them with algorithms such as gradient descent?

I mean, [question 3] is it correct to expect that identified $\gamma$ would account for a global effect of the entity $j$ while $\beta_{j}^{p_1}$ and $\beta_{j}^{p_1}$ account respectively for the contribution of group $p_1$ variables and group $p_2$ variables?

New contributor
sokha is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
  • $\begingroup$ Unless I am loosing something, the parameter (\gamma) is not identifiable. I mean that there is no way do splitt the effect of the variables $x_{ij}^{p_1}$ and $x_{ij}^{p_2}$ between the multiplicative parameter $\gamma$ and the coeficients $\beta_{ij}^{p_1}$ and $\beta_{ij}^{p_2}$. Answer 1) I am not saying that it is useless, but this parameter can not be estimated in such model. Answer 2) No! For the reasons explained above. Note that minimize the MSE you may able to estimate $\alpha_{ij}^{p_1}=\gamma_{ij}\times\beta_{ij}^{p_1}$ and $\alpha_{ij}^{p_2}=\gamma_{ij}\times\beta_{ij}^{p_2}$, b $\endgroup$ – DanielTheRocketMan Feb 14 at 14:34
  • $\begingroup$ The model with $\gamma_{ij}$, $\beta_{ij}^{p_1}$ and $\beta_{ij}^{p_2}$ has too many parameters - and they are not unambiguously defined! Just imagine that these parameters are the best estimates. Then if you multiply $\gamma_{ij}$ by arbitrary constant $C$ and divide $\beta_{ij}^{p_1}$ and $\beta_{ij}^{p_2}$ by the same constant, this parameter set will give you exactly the same result. So you need to decrease the dimension of parameter space here so that the best estimate is just one and unique. $\endgroup$ – Curious Feb 14 at 16:06
  • $\begingroup$ Are you sure you intended to include the $i$ subscripts on the $\beta_{ij}$? For $n$ observations you are positing a model with $np$ parameters, which is far more than could ever be identified. Could you therefore explain what you mean by a "regular" linear model? $\endgroup$ – whuber Feb 14 at 16:15
  • $\begingroup$ I'm sorry for the wrong notation. I meant $\beta_j$. So it's about a total of $p\frac{p}{2}$ parameters: one parameter for each variable ($\beta_j^{p_1} $ and $\beta_j^{p_2}$) and one every two variables $\gamma_j$. $\endgroup$ – sokha Feb 14 at 16:48
  • $\begingroup$ Your use of "parameter" and "variable" is quite confusing, because you appear to refer to parameters as "variables"! I count only $p + p/2$ parameters among all the various gammas and betas (not $p\frac{p}{2}$). It's still unclear what you're trying to ask, because you simply could absorb your $\gamma_j$ into the definitions of the $\beta_j,$ leaving you with a standard multiple regression model. It's crucial to get your notation correct when it is the only means you are using to explain your problem. Maybe you could explain it in some other way? $\endgroup$ – whuber Feb 14 at 17:06