Interaction between a categorical and a quadratic continuous variable I have three main hypotheses:


*

*Independent variable $X_1$, which is a categorical variable of two levels (A and B), may have an effect on the response $Y$.

*Independent variable $X_2$, a continuous variable, may have an effect on the response $Y$.

*The effect of $X_1$ over $Y$ is smaller in lower and upper values of $X_2$, but higher in central values of $X_2$. It is like a quadratic response of $X_2$ over the effect of $X_1$ in $Y$.
Is it possible to express it in a single linear model? Something like:
lm(Y ~ X1 * (X2 + I(X2^2))

 A: Let's start by making $X_1$ a dummy variable (=0 for category A and 1 for category B). The most general version is
$$
\begin{split}
\mu  = & \beta_0 + \beta_1 X_2 + \beta_2 X_2^2 +  \\
       & \beta_3 X_1 + \beta_4 X_1 X_2 + \beta_5 X_1 X_2^2
\end{split}
$$
or equivalently we can say that the intercept, slope, and quadratic term differ between group A ($\beta_0$, $\beta_1$, $\beta_2$) and group B ($\beta_0+\beta_3$, $\beta_1+\beta_4$, $\beta_2 + \beta_5$).  The expected difference between A and B (the "effect of treatment B") is exactly $\beta_3 + \beta_4 X_2 + \beta_5 X_2^2$, so if $\beta_5<0$ your statement (effect of X_1 is small for low $X$ and high $X$ and largest for intermediate values) is reasonable.
This general quadratic model can be expressed as ~X1*(1+X2+I(X2^2) or ~X1*poly(X2,2) in R's (Wilkinson-Rogers) formula language.
One thing to be careful of is that for sufficiently large or small $X_2$ and large (negative) $\beta_5$, $X_1$ could have a strong effect in the opposite direction, so "smallest effect for intermediate values" is somewhat dependent on the context (by "smallest effect" do you implicitly mean a positive effect?) and range (how far can $X_2$ extend towards low and high values)?
