I have a linear model with one continuous dependent variable ($y$) and one explanatory variable ($x$). The model: $y = b_0 + b_1 * x$; now I and need to add a categorical feature to it. The problem is that training for each category should result in its own curve ($b_0, b_1$ pair). Each category is a different process. So I cannot just use a linear regression and add just add $b_2*x_2$.

I'm tempted to use the simplest way: train a separate model for each process. But my feeling is this might fall in one of well-known problems as a single model. The actual problem is in ML field.

  • 1
    $\begingroup$ Let the categories be $x_c \in \{1, 2, 3\}$ then you could define $x_1 = 1$ if $x_C = 1$ and $x_1=0$ otherwise, $x_2= 1$ if $x_c = 2$ and $x_2 = 0$ otherwise. then use $y=\beta_0 + \beta_1 x+\beta_2 x_1 + \beta_3 x_2$. Would that be an option? $\endgroup$ – dietervdf Jan 15 '17 at 2:47
  • $\begingroup$ For each $x_c I need continuous observable $x and a separate both $b_0 and $b_1. In your example only one $b_x coefficient will be added to the sum for each $x_c. Utilizing your idea I can say: for $z_c from (1, 2, 3), if $z_c = 1 $\endgroup$ – VladimirLenin Jan 15 '17 at 3:09
  • 1
    $\begingroup$ in my proposal the intercept will be different for each $x_c$, since when the category is for instance $2$, then the intercept will be $\beta_0 + \beta_3$, while if the category is $x_C=3$ then the intercept equals $\beta_0$. If you want different $\beta_1$ then you would need to add interactions into the model, since the category would change the value of the $\beta_1$ coefficient. $\endgroup$ – dietervdf Jan 15 '17 at 3:14
  • $\begingroup$ Are you building the model in R? $\endgroup$ – dietervdf Jan 15 '17 at 3:14
  • 1
    $\begingroup$ I'll construct an expanded answer, just a sec :) $\endgroup$ – dietervdf Jan 15 '17 at 3:22

Let's say your continous variables are the outcome $Y$ and the predictors $X_1$. Now consider a categorical variable $X_C\in \{A,B,C\}$ then you could define the following variables:

$$X_A = \begin{cases} 1 & \text{if } X_C = A\\ 0 & \text{otherwise}\end{cases},\qquad X_B = \begin{cases} 1 & \text{if } X_C = B\\ 0 & \text{otherwise}\end{cases}$$

Next you could consider the model:

$$Y_i = \beta_0 + \beta_1 X_1 + \beta_A X_A + \beta_B X_B + \beta_{1A}X_1\cdot X_A + \beta_{1B}X_1 \cdot X_B + \epsilon_i$$ which would fit your needs, since

  • Let $X_C = C$ (the reference category)

    Then the model would be: $$Y_i = \beta_0 + \beta_1 X_1 + \epsilon_i$$

  • Let $X_C = A$

    Then the model would be: $$Y_i = \beta_0 + \beta_1 X_1 + \beta A + \beta_{1A}X_1 +\epsilon_i= (\beta_0+\beta_A) + (\beta_1+\beta_{1A}) X_1+\epsilon_i$$

  • Suppose $X_C = B$ then the model would be

    $$Y_i = (\beta_0+\beta_B) + (\beta_1 + \beta_{1B}) X_1+\epsilon_i$$

I think this would fit your needs?

  • $\begingroup$ I think this what I need. Could you help with a common name for this device in R/statistics, if any? Btw, it looks it should be $... = (B_0 + B_A) + (B_1 + B_1a) * X_1 + e_i$ for $X_C = A$. $\endgroup$ – VladimirLenin Jan 15 '17 at 3:53
  • 2
    $\begingroup$ woops, typo :), what do you mean by common name for this device? You can implement this rather easy in R by applying as.factor() on the categorical variabel $\endgroup$ – dietervdf Jan 15 '17 at 4:01
  • $\begingroup$ I mean the name for this should be "non-linear regression"? $\endgroup$ – VladimirLenin Jan 15 '17 at 4:12
  • 2
    $\begingroup$ It is still categorized as linear regression since these are interaction terms. $\endgroup$ – dietervdf Jan 15 '17 at 4:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.