9
$\begingroup$

I'm working on a project that uses genetic data. The dataset has thousands of predictors that are all binary (does the person have a certain letter in a certain position in their genetic code: yes/no). I'm trying to do regression using those predictors, trying to predict the height of the person.

Because each predictor can only have two values, I don't see how there can be any non-linear effects other than through variable interactions.

Is there any non-linearity here that I'm missing?

I'm also thinking about something like a tree-based model (say, random forest). I understand how a tree-based model could outperform a linear regression by assigning different effects (different betas) to certain predictors depending on the value of other predictors. But I would call this "capturing variable interactions". Is there any argument to say that a tree-based model is "capturing non-linear effects" when all the predictors are binary?

And another, related question: What would be the effect of using, for example, different SVM kernels?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Who told you it was nonlinear? That can be a good place to start. Genes interact. Also, when multiple genes are triggered they can have multiplicative impact on the outcome. So gene A makes response to x, and gene B makes response to y, but A and B makes response to x*y or tanh(x)*y or such. $\endgroup$
    – EngrStudent
    Commented Jan 11, 2021 at 15:31

1 Answer 1

Reset to default
3
$\begingroup$

Aside from interactions, which could be vitally important, I do not see it, either.

Let's consider two features that are $0/1$ binary variables.

$$ x_{i1}\in\{0,1\}\\ x_{i2}\in\{0,1\}\\ $$

Now consider some functions $\phi_i:\rightarrow\mathbb R$ of those variables.

$$ \phi_1(x_{i1}) \in\{a_1, b_1\}\\ \phi_2(x_{i2}) \in\{a_2, b_2\}\\ $$

No matter how nonlinear the $\phi_i$ are, the new features are linear/affine transformations of the original variables. You do not get any kind of bending, curving, or discontinuity of the regression by using these new variables. Where you might be able to say there is something nonlinear happening is when it comes to interactions. $\hat y = \hat\beta_0 +\hat\beta_1x_1 + \hat\beta_2x_2 + \hat\beta_3x_1x_2$ is a linear model, sure, but it is not linear in the data. A tree-based model might be able to pick up on this without being explicitly told to look for an interaction, while a generalized linear model will miss it unless you set the $x_1x_2$ interaction as a feature.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.