linear regression - underfitting with one hot encoding

Question

I am trying to fit a hypothesis to my data in order to predict the duration for a certain event.

My data is strictly categorical and I used one-hot encoding for all features.

After using one-hot encoding the dataset now has the dimensions 20.000x414.

First I tried linear regression with regular gradient descent. However, both training and cross validation error is large.

I concluded that this is a case of high bias (underfitting).

Now I want to try a polynomial of higher order but I think that it would not make any difference because all of the features are either '1' or '0'.

If I use the square or the cube of those features they will still be '1' or '0', right?

Is there something else I can try in order to deal with the high bias?

Answer 1

I concluded that this is a case of high bias (underfitting).

This can be checked. Suppose you train your dataset on increasing-sized chunks of your data, and test on some fixed-sized chunk you left out, then plot the train and test errors as a plot of the size of the train chunks.

1. High bias will appear as the error decreasing to some level and staying there.
2. High variance might appear as a large gap between the train and test errors.

If this indeed looks like high bias, you could try random forests, for example, which might find interaction patterns between the features (binary or otherwise). You might find XGBoost, in particular, convenient for use.