I am trying to fit a hypothesis to my data in order to predict the duration for a certain event.

My data is strictly categorical and I used one-hot encoding for all features.

After using one-hot encoding the dataset now has the dimensions 20.000x414.

First I tried linear regression with regular gradient descent. However, both training and cross validation error is large.

I concluded that this is a case of high bias (underfitting).

Now I want to try a polynomial of higher order but I think that it would not make any difference because all of the features are either '1' or '0'.

If I use the square or the cube of those features they will still be '1' or '0', right?

Is there something else I can try in order to deal with the high bias?

  • 1
    $\begingroup$ You can also check the interactions between the categorical variables. $\endgroup$
    – Firebug
    Commented Nov 6, 2017 at 10:34

1 Answer 1


I concluded that this is a case of high bias (underfitting).

This can be checked. Suppose you train your dataset on increasing-sized chunks of your data, and test on some fixed-sized chunk you left out, then plot the train and test errors as a plot of the size of the train chunks.

  1. High bias will appear as the error decreasing to some level and staying there.
  2. High variance might appear as a large gap between the train and test errors.

If this indeed looks like high bias, you could try random forests, for example, which might find interaction patterns between the features (binary or otherwise). You might find XGBoost, in particular, convenient for use.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.